We have a Hyper-V cluster connecting to an Equallogic PS6000e using CSV over iSCSI and recently have begun to setup Hyper-V virtual machine protection.
The issue we are seeing involves the backup timing out (VSSADMIN list writers shows time out) and event logs show iSCSI cannot connect to target volumes. When I look at the iSCSI initiator in Windows (2008 R2) we see all targets communicating successfully. The HIT tools are installed on each Hyper-V host and we have configured the iSCSI targets to use CHAP authentication and IP restriction for the iSCSI network.
If I disable the Equallogic VSS writer (DPM attempts to use this as a hardware VSS writer) then backup of Hyper-V virtual machines succeeds using the default software Hyper-V VSS writer. Our option is to use only IP restrictions for iSCSI access but would we need to provide the DPM server with an interface on the iSCSI network.
What I would like to know is what are the adverse effects of disabling the Equallogic VSS writer using the eqlvss /unregserver command on each Hyper-V node and reverting to the Hyper-V software VSS writer (other than performance impact).Is there a component of the Equallogic SAN on a windows environment that expects the Equallogic VSS writer to be functioning?
Not within DPM. This is known as an "Off Host Backup" or "server less backup" Programs such as Backup Exec do. (Called ADBO) Netbackup and Commvault also support this, but not DPM. DPM can't just "login" it has to be told the IP address and complete IQN name to use.
Generically, to do this, the backup sw, calls Volume Shadow Service (VSS) to quiet the filesystem and flush cache. When VSS goes to do that, if our HW provider is there, it will connect to the vss-control volume on the array, select the volume, create snapshot and return (in XML format) the volume name and IP address. This information can be used to tell the CLI version of MS iSCSI initiator (msiscli) where to, and what volume to log into. DPM lacks this capability.
Also MS changed the XML format in W2K8, so the backup server must be same OS as server being backed up. W2K3 or W2K8, you can't mix them for off host backups.
So DPM would have to build software to handle this processing of the XML file and tell the iSCSI initiator to login. Then put the snapshot offline and delete it when done. So not a simple task by any means.
Social Media and Community Professional#IWork4DellGet Support on Twitter - @dellcarespro
Addendum: Essentially I am looking for best practise from Dell as to the configuration in this scenario.
Assuming you checked the usual (switch configuration, iSCSI disk timeout values), you should open a support case. We can then look at the ASM logs to see what else might be causing this.
The iSCSI disk timeout setting are in a document called “iSCSI Initiator and Operating System Considerations” located on the same page of the Firmware download you currently have installed on your arrays.
The switch configuration’s are listed here: en.community.dell.com/.../3615.rapid-equallogic-configuration-portal-by-sis.aspx
The most important part of the switch configuration is to enable flow control, and Jumbo (if the switch can support both, if no flow control is the most important).
You should also review the "Cconfiguration guide" (look for the link at the top of the page, a lot of good info here): en.community.dell.com/.../2632.storage-infrastructure-and-solutions-team-publications.aspx
Regarding the writer, it’s not recommended to disable the eqlvss writer if using ASM. I’m not certin,but I believe you end up with a “Unsupported VSS application writer” icon and warning in ASM.
Regarding best practices, these are listed in “TR1045-Microsoft Hyper-V ProtectionUsingASMME_ISO.pdf” located on the support site (support.equallogic.com) and “TR1043-HyperV-PSSeriesArray_BP_2.0.pdf” (just go to the Resources> Tech Report Section and do a page search for “hyper” you should see both documents under the Microsoft Windows section – they are named slightly different in the list though, the first one is “Microsoft Hyper-V Protection Using Auto-Snapshot Manager/Microsoft Edition” and “Deploying Microsoft Hyper-V with PS Series Arrays”)
You didn’t mention the backup software you are using.
Also, post the FW version and the HIT version you have installed.
Follow me on Twitter: @joesatdell
You can run DPM w/o the EQL HW provider. Typically this condition is caused because, with the EQL HW provider installed VSS creates a snapshot on the array instead of the unused local disk space on the volume being backedup. However, DPM, won't log into that snapshot so it can't "see" the snapshot the VSS references.
The one benefit of a HW provider is that up to 3 backups can run at once, with the default SW provider only one can.
Here's a list of some common fixes that are needed to get DPM to work better.
A virtual machine online backup fails in Windows Server 2008 R2 when the SAN policy is set to "Offline All"
Unable to backup Live Virtual Machines in Server 2008 R2 Hyper-V
On a Microsoft Windows Server 2008 R2 Fail over cluster with a Hyper-V guest with many pass-through disks, the machine configuration may take some time to come online
Computer randomly stops responding after you use the VSS software provider in Windows Server 2008 R2 or in Windows 7
CSV LUNs fail if you use a VSS hardware provider to back up virtual machines on a Windows Server 2008 R2-based cluster
The Cluster service stops responding if you run backup applications in parallel in Windows Server 2008 R2
MPIO does not log path failure events in the event log on a Windows Server 2008 R2-based computer
MPIO does not remove a disk that is on a failed path in Windows Server 2008 R2
The iSCSI initiator may show a incorrect MPIO path state
Some I/O requests to a storage device fail on a fault-tolerant system that is running Windows Server 2008 or Windows Server 2008 R2 when you perform a surprise removal of one path to the storage device
The TCP Chimney Offload feature fails on all network adapters in Windows Server 2008 R2 or in Windows 7 if you disable or change the properties of a network adapter
The Microsoft iSCSI Initiator may fail to login to Favorite Targets after the Initiator Name is changed on Windows Server 2008 and 2008 R2 or Windows 7
Description of the hotfix rollup package for System Center Data Protection Manager 2010: March 2011
Description of hotfix rollup package 6 for System Center Data Protection Manager 2010: July 2012
Description of Update Rollup 2 for System Center 2012
DPM protection agent service may stop responding on Exchange Server 2010 servers that are protected by System Center DPM 2010
System Center Data Protection Manager 2010 Hyper-V protection: Configuring cluster networks for CSV redirected access
With regard to this section:
"You can run DPM w/o the EQL HW provider. Typically this condition is caused because, with the EQL HW provider installed VSS creates a snapshot on the array instead of the unused local disk space on the volume being backedup. However, DPM, won't log into that snapshot so it can't "see" the snapshot the VSS references."
Do you know how I can get DPM to read into the VSS snap on the array. Is it as simple as configuring a network interface on the DPM server (I.e. a DPM backup network) that can access the iSCSI network or will I still have issues with CHAP authentication for the target LUN's?
The backup software we're using is DPM (Microsoft's Data Protection Manager 2010). We're not using the ASM features on the array unless this is used by Performance or Capacity load balancing - but I can't see reference to this in documentation.
The easiest path is for me to disable the eqlvss writer on the Hyper-V nodes and I've asked Don a question below about how to enable DPM to hook into the target LUN where the snapshot would be stored when using the Equallogic VSS writer.
I work regularly with the DPM team at Microsoft and will request an enhancement and see where it goes.
If you get an answer back, please let us know. That would be a fantastic addon. Such off host are faster and put allot less stress on network and servers.