I have a Equallogic PS6000XV running VMware 4.0 in a cluster, i have 3 hosts running about 18 VM's right now and im wondering what is the correct way or the "best way" to present new volumes to the VM's. I am able to give them access by RDM inside of ESX, or i can use the iSCSI inside windows. I am unable to find any documentation that says which way is better than the other.

I currently have 4 nics dedicated to the vmhba for iSCSI, with MPIO enabled on each hosts per the dell video. what is the best way to procede for exchange, sql, and general file sharing practices? i was told to add a second nic to each server 1 for network and 1 for iSCSI connectivity and run it that way. i would like to take advanced of the auto snapshot manager for these products, is it possible to do this with out the iSCSI iniciator and use RDM? No one has been able to give me an answer with backing for either direction.

There is no "preferred" or "best" way to connect additional volumes. It depends allot of the applications.

RDM's have limits. They are limited to 2TB, and with EQL arrays RDM's cannot use the advanced features of the Host Integration Toolkit . (application consistent snapshots, transportable snapshots, app consistent snapshots, enhanced MPIO, etc)

So for volumes not supporting or needing the advanced features, RDMs are fine.

Remember when using VMDK's or RDMs, to create a new Virtualized SCSI adapter for them. This provides much better I/O performance

For the guest initiated iSCSI traffic, ideally you would have two (or more) dedicated NICs for that traffic. There would be TWO VM portgroups into one vSwitch. The two physical NICs would be aligned with one active NIC and the other standby. The for the other VM portgroup, switch them around.

Same idea as when you setup MPIO for the VMkernel ports, but here you use STANDBY vs. UNUSED with Vmkernel ports.

Vmware calls this kind of configuration "Storage Direct", since the VM is totally responsible for managing the iSCSI & network connections to the array.

Hope this helps.

-don

Are you using RDM now of VMFS? I mean, you should continue doing the way the current disks are setup, would be my suggestion.

Don,

Thanks for the reply. I would like to use the advanced features, i just wanted to make sure that i was not changing the performance by using RDM. I have also just upgraded to the new 5.0 firmware, has some nice features and a new look.

Here is what i think i need to do. My current server has 12 Nics in it. Im going to take 2 or 3 of these and create a new vswitch that has jumbo frames enabled on it, create 3 new kernel ports (1 per nic) and add jumbo frame support on the nics and tie and ip address to them. I was going to make them all active unless you think there is a reason for active / standby. How should i handle the OS nics. just 1 for network and 1 for iSCSI or multiple for iSCSI?

i used this video to setup our EQ http://www.equallogic.com/resourcecenter/video.aspx?id=8455, and i figured i could adapt this for the iSCSI data volumes instead of the guest OS.


Thanks again.

Bill

"Are you using RDM now of VMFS? I mean, you should continue doing the way the current disks are setup, would be my suggestion."

Its hard for me to tell because at this point i do not really know the difference. The drives are setup inside vcenter as a RDM. I do not have any production servers, this is all testing for the next few weeks then its time to go live. Any suggestions one way or another?

Vmware bills the RDM feature as the best of both worlds so I don't think you encounter any issues.

The difference between RDM and VMFS is, VMFS (VMware File System) is a volume formatted by ESX. It's a container for multiple VMs and each "disk" that the VM sees is actually a file stored on that volume, called a "Datastore" In this case the entire process from iSCSI handling to file handling is done by ESX.

An RDM (Raw Device Map), is where VMware handles the network/iSCSI side for you. The RDM is formatted with the filesystem from the Guest.

Lastly:

Storage Direct is where the VM has its own iSCSI initiator and handles all the ISCSI functions itself. VMware is just a network conduit to pass traffic.

So:

VMFS == Everything is fully virtualized (handled) by ESX

RDM == Part of the process is handled by ESX part by Guest OS

Storage Direct == Everything is handled by Guest

RDM's simply things but also go against the maximum number of volumes Vmware supports and are limited to 2TB. Also you can't use our Host Integration tools with RDMs.

Storage Direct is the opposite. Vmware doesn't see it as a volume, no filesize limit, and works with all our tools. If you already have EQL volumes with physical servers, then doing P2V is dead simple. Since all you have to virtualize is the "C:" drive.

So what's "best" depends on what you need the volumes to do. If you want all our tools, then Storage Direct.

Does that help?


-don

This thread shows that there is a need for more information on Storage Direct with EqualLogic SANs. I guess I would look to EQ to write a technical report, that not only shows how to setup Storage Direct properly, but when it is recommended to use it. Everything I have seen and heard says that for best performance from Windows guests, particularly those where application-aware snapshots for Exchange or SQL are desirable, then use storage direct. Our hosts have eight NICs, two for the VMware software iSCSI initiator, and two for guest iSCSI. At this point any guest that requires a data volume of any large size uses storage direct. Setting it up is not as straightforward as using a VMDK, so it would be great if EQ would write a tech report to help educate its customers in how to get the best performance from their SAN.

That's my 2 cents, anyway.

Hello,

I agree with you on the white paper, that in fact is in the works.

To bring it down to the barest level, if you want Host Integration Toolkit features like, application consistent snapshots/replication. The ability to tightly integrate in to MS SQL/Exchange, off-host backups, then Storage Direct.

The Storage Direct setup is pretty straightforward.

Two VM networks into one vSwitch.

The vSwitch will have two Physical NICs.

One NIC will be active for one VM network and the other NIC standby.

Reverse it for the other VM network. The NIC that was standby becomes active for this VM network and the other standby.

that's it.

Sorry I can't figure out how to attach images here or I could show you a sample from my setup.

-don

Photos would indeed be helpful, however I setup my switch and port groups like that some time ago. I'm sure there are plenty of new EQ customers that would appreciate this info, and I'm glad to hear that the TR is in the works. I use TR1049 , specifically the script in the appendix, to help automate the setup of our hosts. A complete guide does make it much more simple.

TR1049 was for the VMware iSCSI side, which is slightly different since you have to set the inactive NIC to UNUSED vs. standby for Guest initiated iSCSI traffic

-don

Good to know, I've been setting them to unused.

Just to be clear:

For VMware ISCSI VMkernel ports the alternate NICs must be UNUSED

For Guest initiated iSCSI traffic you would want the NICs to be STANDBY

-don