This in-depth, technical discussion on Internet SCSI (iSCSI) security and best practices provides a part 2 follow up to the Dell iSCSI and Virtualization chat on 6/30/2009. Links to several articles on general iSCSI misconceptions (iSCSI security, performance, enterprise applications, and so on) are provided here.

Guest experts participating in the discussion include:

  • Darren Miller
  • Eric Schott
  • Keith Swindell
  • Robert Winter

Technical Community - Background Reading

iSCSI adoption

iSCSI virtualization

iSCSI security white papers

iSCSI and databases

iSCSI offload characterization

Chat Transcript

Dell-JeffS Hello folks. Wow, early turnout already!
ceri Evening. I can never remember if it's at 8 or 9 local time ;-)
Dell-JeffS Hi Ceri, it looks like 9 your time...we start 30 minutes from now *officially*
ceri That'll be it. No bother to just get here early and wait for everyone else ::blush
Dell-ScottH Who's got the fireworks?! I'm ready!
JasonPowell /me rolls eyes :-) but you did get my attention
Dell-ScottH The 4th is my fav! I would love to pack cherry bombs in an HP server and do it some good!
JasonPowell Just got an order for an EqualLogic PS5000e—w00t!
Dell-JeffS Nice, and we haven't even started yet!
JasonPowell Lol, been working with this customer a while
Dell-ScottH Very nice, a pallet of them, right?
JasonPowell /me dreams; that would be a huge hit to my debt snowball
Dell-JeffS We have four EqualLogic PS5000s in the lab. An EqualLogic PS6000 with an SSD should arrive soon. Heard today we might get to order an EqualLogic PS4000
JasonPowell I heard some "interesting" performance stats on the EqualLogic PS4000 when joined to EqualLogic PS5000/PS6000
JasonPowell I'd like to do some testing on an EqualLogic PS4000 too
JasonPowell But I'll wait for you guys to do all the work for me ;-)
Dell-JeffS Take a number :)
JasonPowell Get those cables traced down from earlier?
Corey In isolating out traffic for iSCSI on the EqualLogic PS5000 and PowerVault MD3000i to a separate physical LAN, we found that none of the Dell switches will handle L3-jumbo. L2 was fine, but it made connectivity somewhat difficult
Dell-JeffS A few of them. Scott, Kong, and I have quite a bit of work to do in the lab. Today's efforts were to get ready for a visit from a Novell SUSE SME on Monday. But we really need to spend some time figuring out where all those cables are going! Years of hardware additions and removals have made some serious spaghetti
Dell-ScottH Oh man, we need a late night/weekend session to clean up the lab and reorganize. I see many vMotions and storage vMotions in our future—yeah, just through some sauce on it!
Esco-n-MyEQL I will take any leftover equipment for my lab at the house
JasonPowell We’re almost done with a complete wiring overhaul
Dell-JeffS Lol, I'll bet.
JasonPowell Color coding wires...labeling each end...etc.
Dell-JeffS Hello Trekkie
Dell-ScottH Yeah, that definitely makes for easier workings in the lab
Dell-JeffS Does that mean you are offering to do ours?
JasonPowell I'd love to come help you guys
Dell-ScottH We have all the cables and different colors and lengths, just need the time; come on down!
JasonPowell You cover my travel and I'm there :-)
Dell-ScottH Everybody seen this? www.youtube.com/watch?v=nrhgpirftjo My manager said, "You are a big dork." :-)
Eric_Schott On Dell switches, I understand PowerConnect 6200 supports L3 and jumbos
Esco-n-MyEQL You Are a big dork
JasonPowell @wantmoore says he has to come too
Dell-ScottH Thanks!
JasonPowell Says “video not available"
Dell-JeffS Make sure and right-click on links
Dell-ScottH Come on down, and we will treat you to a real burger from Mighty Fine...and you can say goodbye to 5 Guys :-)
Dell-JeffS Otherwise, you'll get booted like Eric and Trekkie probably just did
Dell-ScottH This one's got cool sound effects :-): www.youtube.com/watch?v=1mpeiper3oa
Dell-JeffS Hey Eric, Trekkie, you ran into one of our chat's features; automatic exit on links. Make sure and right-click on links
JasonPowell Just re-tweeted for this chat
Dell-JeffS Thanks!
Dell-ScottH Thanks for the RT love!
erson Hi all
Dell-ScottH Hey Erson!
ceri Hi Erson
Trekkie Does the chat thing not like it when you go to another tab too? heh
erson /me is watching PBS Frontline: “Breaking the Bank”
Dell-ScottH That was a good episode
Esco-n-MyEQL Scott, I tried to send you a text but it failed...did you change numbers?
ceri Trekkie, yes. Don't open another tab in the same window, or you get booted. Did that about three times on Tuesday
Dell-ScottH Nope, my cell # is same
Esco-n-MyEQL My corporate BlackBerry must not like your phone
Dell-ScottH It's an iPhone :-) Michael is probably blocking all my messages :-)
Esco-n-MyEQL He knows you well!
prickett Is there an audio number, or will there be audio on this page?
Dell-ScottH No audio, this is a text-only chat session. Jeff will kick it off at the top of the hour and introduce the Dell experts we have on the chat, give out a few links, and then chaos begins :-)
Esco-n-MyEQL I will play the role of a "non" expert on this chat
erson Hmmm, can I install an EqualLogic multi-path I/O device-specific module on Windows Server 2008 Core?
Darren_Miller Erson, we support Windows Server 2008 Core with our Host Integration Tools v3.2
Dell-ScottH Is there a link to that? Or is it on the CDs that ship with the box?
JasonPowell Hit kit is on the EqualLogic support site
Darren_Miller It's on the CD and/or EqualLogic support site
JasonPowell EqualLogic.com/Support
Dell-JeffS Okay, it’s that time. Hello everyone and welcome to the Dell TechCenter chat on iSCSI security and best practices
Dell-JeffS Just a few housekeeping items: first, if you see a link, right-click it; otherwise, you'll likely get bumped out of the chat and have to rejoin. This is an informal chat; feel free to ask your questions at any time
JasonPowell Informal++
Dell-JeffS And should you find yourself bumped out, you can always take a look back at the previous dialog by clicking Action, Recent Room History
Dell-JeffS Okay, we have a number of folks here to take your questions today. Eric Schott is the director of product management for EqualLogic. Keith Swindell and Darren Miller are product managers
Dell-JeffS Scotth and I are on the Dell TechCenter team. Does anyone have any questions they'd like to start with?
ceri I think the initial RFC for iSCSI said "you really should use IPSEC" in not so many words. Is this still the best option for security, and how widely is it supported?
Mike_C If I have multiple servers on the same subnet as my EqualLogic array for shared storage I assume that these servers can see each other on my iSCSI network. What is the best method to lock these servers down to prevent access?
Keith_Swindell Ceri, IPSEC is not essential for good security, and is not widely implemented
ceri Keith_swindell, so, dedicated networks then?
JasonPowell We just limit each volume on EqualLogic to the IP address(es) of the VM attaching to it
Keith_Swindell Mike_c, yes they could "see" each other, but unlike in FC SANs, this is not a recipe for disaster. In general, you will have no more danger from sharing the SAN network than you do on the data side where you interact with the end users. In fact, since the SAN is typically isolated from the users, it is actually less of an issue
JasonPowell Our SAN network is 192.168 and LAN is 10
Keith_Swindell Ceri, the SAN should always be separated from the front end data network, either with dedicated switches, or at a minimum, with a separate VLAN
Mike_C True for the iSCSI volume access, but what prevents me from access to the Ethernet port of the iSCSI connection from server to server. Would a firewall be used?
martin_huber The documentation provided by EqualLogic/Dell is bit unclear when it comes to VMware ESX and iSCSI configuration for EqualLogic SANs. There are number of papers stating that to improve performance one should "use the NIC teaming feature to utilize multiple NICs on your VMkernel switch; this will establish a unique connection for each volume, which the EqualLogic array can then load balance." I have found that using multiple NICs teamed accessing multiple volumes on the SAN still utilized one NIC. Is there a paper you can point me to that describes how (step by step) to configure this correctly as well as performance papers showing the benefits of teaming?
erson Are there any significant differences in performance and functionality between a dedicated iSCSI HBA and the Broadcom bmc5709c (used on an 11th-generation Dell PowerEdge server)?
ceri Keith_swindell, sure but you don't recommend anything more than that at present?
Eric_Schott For SAN traffic, MPIO is generally a better solution. The challenge in VMware 3.5 is MPIO is only active/passive. In vSphere 4, it is active/active
ceri Erson, best thing you get out of an iSCSI HBA is the ability to boot over iSCSI, I think. With the QLogic cards, you can specify the volume to boot from in DHCP too
Eric_Schott You are correct that teaming will only use one port for a given connection/session
erson Ceri, you can boot from iSCSI with pretty much every decent server NIC these days
Corey We have found that the iSCSI offload on smaller modern servers make no significant difference. In the range of single-digit CPU differences. Older servers or very heavily I/O-loaded servers matter a bit more, but for us jumbo frames are about equal to iSCSI offload in general. Jumbo frames are less $ and simpler to implement
Eric_Schott For Erson, what iSCSI HBA are you comparing to the Broadcom?
erson QLogic seems to be most common
ceri Erson, really? I've completely failed to see that anywhere. Hmmm…
martin_huber So, the statements in the EqualLogic papers are incorrect? I can not increase performance with teaming. Would that be a correct statement?
Keith_Swindell Ceri, generally no. You can put in ACLS to prevent access to the SAN VLAN if you wish to prevent other systems from connecting to the storage, and some service providers will separate each client's servers into their own VLAN and route with a high-performance L3 switch to the storage VLAN in conjunction with ACLS, but the vast majority of customers don't have these sorts of requirements
Corey Our tests were Broadcom iSCSI offload, Intel, and Broadcom GbE NICs and QLogic iSCSI HBAS
erson I can't use HBAs on my PowerEdge M710s, but I'm interested in knowing if I'm missing something (other than to shell out for dedicated HBAs)
Darren_Miller Martin, which paper are you referring to?
erson Corey, what was your conclusion?
Eric_Schott Erson, Broadcom will outperform QLogic. As others have stated at 1GbE many software initiators on current servers are good. Note Broadcom is only supported on Windows today with full offload, so it is not always a choice
martin_huber "VMware ESX Server 3.x with An EqualLogic PS Series Group" V1.29
Eric_Schott Martin_huber, teaming can help performance, but it is in aggregate (i.e., connections to multiple volumes). For a single connection to a volume, teaming will only use 1 port
martin_huber I'm using multiple volumes and only see traffic going across one NIC
erson /me creates tabs in the same windows as the chat with no problems using IE 8. What browsers did Ceri and Trekkie use?
Eric_Schott This was discussed recently in some blogs we did: http://virtualgeek.typepad.com/virtual_geek/2009/01/a-multivendor-post-to-help-our-mutual-iscsi-customers-using-vmware.html
Dell-JeffS Right-click those links
ceri Erson, I am using Webkit
erson /me left-clicks and the link opens in a new window
martin_huber Thanks!
Eric_Schott Martin, on traffic on one port, I assume you have all the array interfaces enabled? On your team you want to check what load balancing is being done: MAC, IP address, etc. You may have to adjust the team balancing policy or adjust addresses
Dell-ScottH Wow, that is a great blog post. Had not seen it before; thanks for linking to it
martin_huber Eric, yep I have played around with different teaming modes; all my NICs are active (there are four in the iSCSI vswitch)
erson What’s the best feature that EqualLogic has that Lefthand does not?
Eric_Schott Erson, tools, no requirement for matched pairs, no requirement to have to use R1 as the fundamental form of data protection
Keith_Swindell Erson, our virtualization technology is also best of class, allowing us to scale both capacity and performance and to have multiple hardware generations interoperate seamlessly
Dell-JeffS Are there any questions on the security aspects of iSCSI implementations?
kerryv20103 Thanks!
erson Is this the chat when you announce that you have certified 2 TB SATA drives? :)
Corey QLogic on older servers (pre-Core2/Tulsa Xeons) was a help as was Broadcom. Similar performance, but QLogic was a bit faster than the older Broadcom. Jumbo with Microsoft iSCSI initiator was plenty fast on later servers and was less trouble to implement, not to mention less $. We only used dedicated NICs for iSCSI, but both the Broadcom and the Intel in NIC-only mode with jumbo were able to provide 1–2 Gb of throughput per host with little effort. The iSCSI SAN was more of a determining factor. Most of our tests were with PowerVault MD3000i, which is a lower-end iSCSI SAN. With MPIO configured paths we could use two NICs simultaneously on single-threaded tests with good results. FC HBAS are still generally faster and had lower overhead, but the FC switches and HBAs make smaller servers very expensive to add to the SAN. iSCSI with these same servers made good sense for the $/performance you get. Since we used MPIO, the teaming issue is not a problem
Corey The *best* security is not to have any non-iSCSI connectivity! ;-) L2 jumbo iSCSI-capable Dell switches are fast and cheap. Not worth trying to secure
kerryv20103 From a cost perspective, is it more or less expensive to secure an iSCSI environment opposed to say FCoE?
erson Do you recommend to use dedicated ports on the EqualLogic for management or to just hook up the management network to the iSCSI network?
Keith_Swindell Kerry, FCoE requires DCB and very high-end switches; iSCSI can benefit from DCB, but doesn't require it, so from a cost perspective, iSCSI will be much more palatable for many customers
kerryv20103 Okay
erson A good thing with iSCSI is that it's just plain old Ethernet, so network security is pretty much the same stuff as you have been doing to secure your network before. What’s DCB?
JasonPowell Hey, what's the traffic difference between jumbo versus non-jumbo? Just rough percentage-wise?
Keith_Swindell Erson, most customers use the iSCSI ports on the arrays for management and also since it is very low overhead. The dedicated management port option is available for those customers that need to segregate management due to business requirements
erson My business requirement is to keep the services up and running so I guess that don't apply to me then :)
monsanto Erson, it looks like DCB is Data Center Bridging: www.ieee802.org/1/pages/dcbridges.html
kerryv20103 In earlier training I was taught EqualLogic iSCSI works best with say Cisco 3750s, but they don't really have great security features. Any high-end Cisco switches that EqualLogic now work better with?
Keith_Swindell Erson, DCB is Data Center Bridging (sometimes referred to as Data Center Ethernet). You can Google it, but the working group is here: www.ieee802.org/1/pages/dcbridges.html
erson Monsanto owned you on that one... :)
Keith_Swindell Essentially, it is intended to make Ethernet lossless, and is required for FCoE
erson Keith, excellent seven-word summery. I immediately got tired when I looked at the IEEE-page that you and Monsanto linked
Corey Kerry, just be sure the high-end switch line cards you pick are not oversubscribed. Most Cisco 1GbE line cards are 8:1. The Ws-x-67xx on the 6500s are not and make great switch/server ports. The 45xx switches are more limited, but the newest supervisors and line cards can do well. The 3750/3560 is the simpler choice, but the software options are more limited
Keith_Swindell Kerry, the Cisco 3750s work quite well, but are not the only good switch out there. With the right I/Os, you can use ACLS with the 3750s, but most people just isolate the iSCSI SAN and restrict access to it from the core of the network. See the tested switch list on SalesEdge. While not all switches that will work are listed, the ones on there are known to be OK to use
erson But what does "work best" really mean? Has that anything to do with performance in 99.9 percent of the cases, or is it just generally more feature-packed Cisco switches that make them to "work best"?
erson Keith, SalesEdge?
Dell-JeffS Hey Erson, either your sales rep or Scott/I can get you the information. SalesEdge is an internal thing
Keith_Swindell Jason, the difference in throughput with jumbo frames versus Standard Frames is quite small for most workloads, often only 5–10 percent, but with large sequential traffic it can be much higher
erson Okay, so it's not something you get access to when you have support-account on EqualLogic.com
Corey Kerry, if you want just L2/L3 bandwidth, 1U switches typically are the best $/performance. Stacking is pretty painless and most of the software features you paid $$$ for in the high-end Cisco world you won’t use with iSCSI. A 65xx chassis costs $10,000/empty slot plus the cost of the line cards. The 3560/3750 cost about the same as the 65xx line cards but without the chassis $$$. The Dell switches for a smaller network have been working fine when you just want dedicated iSCSI L2 switches. Each of our iSCSI SAN networks are flat L2 anyway, so any vendor-managed switch works. Higher-end ones have better management and such, but...
JasonPowell Keith, I was thinking it was a more drastic difference in traffic...hmmm…I have a customer talking about doing live editing of video files on EqualLogic; their current solution does not do jumbo frames
kerryv20103 All your comments are great. I was really looking for ammo when dealing with customers who already have invested in Cisco high-to-medium-sized switches, and I want them to integrate an EqualLogic PS5000e/PS5500.
Darren_Miller Jason, video tends to be more on the large sequential side of the house; jumbo frames would help them out. Are they having issues with their current environment?
Keith_Swindell Erson, no, sorry, internal document. Our policy has always been to endorse open standards, and we will try to support most enterprise-class switches, but only have resources to formally test a subset of them. Not being on the list does not mean "won't work"
Corey Note that jumbo on many vendors’ switches are a mixed bag depending on the ASICs on the box and the software features. Verify everything before you configure or promise
kerryv20103 Some customers only use 3750s for user access and not for storage services, so they would need to modify their switch infrastructure to accommodate this
erson Keith, no problem. I'm going to use PowerConnect 6200/6300-series for my SAN network, and I'll bet those are certified
JasonPowell Darren, yes, editors are complaining that their current SAN is slow
Keith_Swindell All things considered, flow control will get you better ROI from a performance perspective. Jumbo frames sound cool, and are nice to have, but not critical for most customers
JasonPowell It's iSCSI...so they are gun-shy going to EqualLogic. They are wanting XSAN (naturally)
Keith_Swindell Erson, Dell does make sure that Dell switches work ::smile
Corey Kerry, iSCSI is really more like FC in many cases, so treating it as a separate LAN is easier. IOPS is more of the problem most of the time than Gb/sec. Spindles are slow, very slow. I call the disk activity light the go-slow light. SATA is storage size, not performance at all. Very low IOPS in general, but every situation is different. It matters more how you fit it than what it is. You can get almost any vendor’s stuff to run fast if you have enough of it! ;-)
Corey A good tailor is more important in the comfort of your clothes than the style of the suit
kerryv20103 ::laugh
Dell-JeffS Any other last-minute questions? That hour went by quickly!
Corey I agree with Keith, jumbo, flow control, MPIO, and GbE on a modern SAN with enough spindles, and there should not be serious performance issues in most cases
Keith_Swindell Jason, with video editing, jumbo frames would be beneficial. We can take this offline and talk about the specifics of your customer's needs next week
JasonPowell Any recommendations on a front end for EqualLogic that makes it XSAN "like"?
erson Jasonpowell, get a can of silver spray paint?
JasonPowell Meta SAN is one product I've looked at
JasonPowell Erson, and an Apple sticker ;-)
erson They are editors...they wouldn't notice any difference :)
Dell-JeffS Well, its 4:00 and I'm sure some of you in the U.S. are ready to start the holiday weekend. Thanks everyone for joining. Excellent questions. A transcript will be posted.
Darren_Miller Jp, you may want to look at 10GbE solutions for your video customer. Much better performance for large I/Os
Corey Hard to beat a SATA/ESATA local drive for scratch performance when video editing. Let the SAN store the finals and the scratch be local. Much faster as latency is lower
Dell-JeffS Thanks to our experts, Darren, Keith, and Eric for all the great feedback today
kerryv20103 Thanks!
Dell-JeffS Have a great weekend everyone!
Corey Performance is a balance of I/O, not the fastest single pipe you can fit. 10GE is great, but the weakest link issue will remain
erson Yeah, thanks a lot EqualLogic guys
Corey Indeed. These are nice to have available...
JasonPowell I have a number of people looking for XSAN alternatives, so I'm trying to figure out how to fit EqualLogic into that space—would be happy to offline with some brain dudes, if needed :-)
erson Okay, getting late here in Sweden; see you all next time
Darren_Miller Jp, send me an e-mail, I can put you in touch with some folks that can help. darren_w_miller@dell.com
Dell-JeffS Jason, we can make that happen. DM me Monday
Darren_Miller Thanks Jeff
JasonPowell And for the record, I moonlight for Vr6 Systems / Alan Hunt...seems like a lot of EqualLogic people know of Alan
Dell-JeffS Okay, I'm out! Spawn, spousal unit and I are heading out for the holiday weekend. Don't catch your lawns on fire with those fireworks!
JasonPowell Darren, I'll send ya e-mail
Darren_Miller Have a good weekend! Sounds good. I'm out too…cheers!