Join a Live Case Study discussion presented by key team members who helped HotSchedules deploy a virtualized data center, disaster recovery (DR), and data protection solution to accommodate steady annual business growth.

Who Is HotSchedules?HotSchedules provides online workforce management and scheduling for organizations in the restaurant, hospitality, and retail fields to streamline employee scheduling, facilitate communication between managers and staff, and reduce costs. In the face of rapid business growth, HotSchedules wanted to enhance the delivery of its software-as-a-service solutions and help ensure maximum uptime. To achieve these goals, Dell Global Infrastructure Consulting Services helped HotSchedules design and deploy a virtualized IT infrastructure based on Microsoft® Hyper-V™ technology.

Advanced High Availability and Disaster Recovery
HotSchedules consolidated servers and implemented advanced high availability and DR with Microsoft Hyper-V and Double-Take® for Hyper-V software.

Working with Dell Global Infrastructure Consulting Services, HotSchedules used a multiple-phase approach for consolidating servers and implementing high availability and DR—key drivers for its data center solution—to help reduce power consumption by 77 percent and software licensing costs by 75 percent.

In addition to consolidating Dell™ PowerEdge™ servers and Dell PowerVault™ storage, HotSchedules implemented the Microsoft Windows Server® 2008 OS with Hyper-V, the Microsoft System Center Virtual Machine Manager (SCVMM) and System Center Operations Manager (SCOM) suites, and Double-Take® for Hyper-V software to protect critical applications and data.

Chat With the Teams That Made It Happen
Join this discussion to hear from HotSchedules, Dell, and Double-Take Software team members about their approach and best practices for implementing:

Phase 1: Capacity planning and consolidation on Microsoft Hyper-V
  • Performance monitoring, good physical-to-virtual migration candidates, and virtual machine scalability in the physical design
Phase 2: High availability, DR, and manageability
  • DR and data protection using Double-Take software
    • Provide enhanced flexibility for protecting the Hyper-V host and individual virtual workload.
    • Present cost-effective protection for virtual and physical workloads.
    • Deploy with auto discover and rapid provisioning features and allow future live DR testing with no production downtime.
  • Manageability using SCVMM, SCOM, and Performance and Resource Optimization (PRO)-enabled management packs
Scheduled Presenters

  • Ray Pawlikowski, president, HotSchedules
  • Matt Woodings, CTO, HotSchedules
  • Brace Rennels, technical marketing manager, Double-Take Software
  • Ray Weinstein, senior global solutions architect, Dell Global Infrastructure Consulting Services
  • Burk Buechler, services technical lead consultant, Dell Global Infrastructure Consulting Services
  • Brent Douglas, Hyper-V SMB bundles lead, Dell Virtualization Solutions Engineering
  • Ranjith Purush, lead engineer, Dell Virtualization Solutions Engineering
  • Kong Yang, specialist, virtualization and power and cooling, Dell TechCenter

Technical Community – Background Reading

Chat Transcript

KongY-Dell Looks like the experts are here. Let's get started
KongY-Dell Any questions from our audience?
erson When did you begin phase 1, and when was phase 3 completed?
Ray_Weinstein Good afternoon all! On behalf of Dell and HotSchedules we welcome you to this Techchat
matt_woodings Phase 1 started June 8 last year, and Phase 2 was completed February this year, and Phase 3 is in testing
Ray_Weinstein Please let us know if you have any question about our highly available Hyper-V solution with Double-Take DR
KongY-Dell Welcome Lance
brennels So I understand HotSchedules was looking for a high availability solution to protect their Dell Hyper-V servers
Matt_Woodings_HotSchedules We were looking for a full DR solution that included Hyper-V and SQL Server
brennels Matt, how many Hyper-V servers did you set up originally for phase 1?
erson With lots of customers depending on your Web site being available at all times?
Matt_Woodings_HotSchedules For phase 1 we had approximately 50 servers of which about 40 were stateless application servers that didn't need replicating, leaving 10 critical servers
Ray_Weinstein HotSchedules currently relies on Hyper-V to host its 380,000-user application architecture
Matt_Woodings_HotSchedules We are online 24/7 365 days. We can't afford any downtime
erson Okay, that sets the baseline :)
brent_douglas How did the need for 100 percent uptime affect the design?
Ray_Weinstein HotSchedules partnered with Dell Consulting Services to design, validate, and then protect the entire environment
brennels So is that why you went looking for Double-Take Software to keep your system for Hyper-V highly available, and did you do any testing before you rolled it out?
KongY-Dell Case Study 1: http://content.dell.com/us/en/enterprise/d/corporate~case-studies~en/documents~966-2008-hotschedules-810002930.pdf.aspx
KongY-Dell Case Study 2: www.dell.com/content/topics/global.aspx/casestudies/en/us/us/fy2010_q1_id1194?c=us&cs=555&l=en&s=biz
brennels Using Double-Take for Hyper-V allows the entire Virtual Host server to be failed over, and there were also some stand-alone SQL servers that Double-Take is protecting as well
Matt_Woodings_HotSchedules Double-Take has a long history of virtually real-time replication over the wire, which made it a perfect candidate for off-site SQL and virtual machines
Ray_Weinstein 100 percent uptime was critical; as defined in the case study, we used Hyper-V host clustering to ensure local availability and then used Double-Take for Hyper-V to provide a complete DR scenario for priority VMs
brennels Flexible enough to support both Hyper-V and stand-alone servers with the same product then
erson Is the Double Take replication asynchronous or synchronous?
Ray_Weinstein HotSchedules can now recover from complete primary data center loss in a matter of minutes
KongY-Dell You can click Action, Recent Room History to read what’s been discussed so far
brennels Double-Take is asynchronous and provides byte-level replication as the changes occur
erson Were there any thoughts on using EqualLogic arrays for replication instead of MD3000i and Double-Take software? EqualLogic replications also being asynchronous
brennels The SQL transactions are written in write order preservation to maintain data integrity and with the built in compression; SQL transactions can be compressed upwards of 80 percent to reduce overall bandwidth requirements
Ray_Weinstein Erson, HotSchedules was a Hyper-V early adopter from March 2008, and at that time the MD3000i was the preferred storage solution
Matt_Woodings_HotSchedules Our database is over 300 GB and replicates with Double-Take over a 10 MB pipe to be within a second or so of the actual running database
brennels I believe that Double-Take has been used in combination with EqualLogic to provide additional options for host-based as well as device-based replication
erson Sounds like the Double-Take software gives you smart replication instead of just dumb replication
brennels Matt, that is great performance; must be the strong Dell architecture :)
brennels It is intuitive; here is some additional information about replication on Hyper-V though
Ray_Weinstein Erson, the Double-Take console is very intuitive and powerful, and we were confident within hours
Matt_Woodings_HotSchedules As the Double-Take representative stated, its buffers write in the order they were written, and only writes them on the target after it’s collected them in the order they were written. This prevents corruption and allows the WAN to be a good transport
brennels Because Hyper-V writes changes to the VHD files, they are written to disk in blocks and then replicated in that order
KongY-Dell For more information on the case study, refer to www.dell.com/content/topics/global.aspx/casestudies/en/us/us/fy2010_q1_id1194?c=us&cs=555&l=en&s=biz
brennels This isn't quite the same when using Double-Take on a non- virtualized server because the changes are written directly to the disk and can be replicated at the byte level
Ray_Weinstein Matt, what was the final physical-to-virtual consolidation ratio?
Matt_Woodings_HotSchedules We consolidated 50 physical servers onto three PowerEdge servers with a peak consolidation ratio of about 20:1
brennels Looks like the case study says there were 50 physical servers that were all virtualized on the three PowerEdge nodes
erson Are there any plans to take advantage of the much improved features of Hyper-V in Windows 2008 R2 that will be released later this summer?
erson 50 VMs on three PowerEdge 2950s
Matt_Woodings_HotSchedules Hyper-V 2 offers some great improvements that most notably is live migration
erson A PowerEdge 2950 can have 64 GB of memory with 8x 8 GB configuration (not cheap). Is that the configuration being used?
Ray_Weinstein Erson, HotSchedules currently has Windows Server 2008 R2 (with Hyper-V 2.0) in production as a Tap customer. Live Migration and CSV has been in production since December 2008
erson Great. We're testing that same setup right now
Matt_Woodings_HotSchedules Having seen live migration in practice, its very impressive
Ray_Weinstein Excellent, Erson
brennels That is a good point. Because you are on the Hyper-V infrastructure you can easily move the workloads between virtual machines using Live Migration
erson Waiting for the PowerEdge M710 to take things into production. When demoing Live Migration in public for the first time they live migrated a VM playing a 720p video...not a single frame dropped
Ray_Weinstein Erson, true. Failover is within milliseconds
Matt_Woodings_HotSchedules I have tested our application and RDP, of which I think RDP was the best test of it. I had someone typing in notepad while we moved their VM onto another server, only a moments pause was seen
brennels There is a link to a blog post on the chat session main page that has an image of the overall architecture, just can't copy and paste it for some reason
brennels Very cool. Matt, how far away is your disaster recovery data center?
erson Brennels, this pdf from Double-Take has the same image: www.doubletake.com/documents/case_studies/hotschedules_dbtkcasestudy_final.pdf. Being from Double-Take I suspect you already knew that :)
Matt_Woodings_HotSchedules We currently have it on a metro Ethernet link 20 miles from the main data center
brennels It's called working it ;)
KongY-Dell @brennels, here's the link that you mentioned: http://userblog.doubletake.com/2009/05/14/dell-double-take-hotschedules-present-hyper-v-live-case-study/
brennels Thanks, Kong
ranjith_purush Does your recovery data center also have an identical hardware setup (PowerEdge 2950s and PowerVault MD3000i’s) as your primary data center?
erson When you click the image on that blog, you get to a page when you can get that pdf I linked above :)
Matt_Woodings_HotSchedules Yes, we have two PowerEdge servers and a PowerVault MD3000i at the DR site
erson Is there a backup solution?
Matt_Woodings_HotSchedules I never keep all my eggs in one basket; I also have backups on the VMs and database on site
Matt_Woodings_HotSchedules We also keep tape backups safely off site also
erson What backup solution do you use for the VMs?
erson We're currently testing System Center Data Protection Manager with much success
Ray_Weinstein Erson, cool, using VSS?
erson Keeping all the VMs backed up on a PowerVault MD3000i
erson Ray, yes
brennels Good solution. I see more and more companies rolling out VSS with off-site archiving
Ray_Weinstein Erson, what are your workloads?
ranjith_purush Erson, what kind of servers do you have in your environment?
Ray_Weinstein (applications)
erson We're going to transition four pretty heavily used SQL servers to Hyper-V currently one is used in testing. We're still waiting for three PowerEdge M710 servers so we're using two PowerEdge 2950s while waiting
brennels Matt, are you planning on migrating any more physical servers to Hyper-V, and what are you planning on using to do the physical-to-virtual conversion?
erson We have about 45 employees with Small Business Server 2008 with plans to migrate to Essential Business Server 2008 when we get the new hardware set up
Ray_Weinstein Erson, cool, are you using a tool to baseline the resource requirements?
Matt_Woodings_HotSchedules I currently have two physical servers left to migrate, which I plan to use Double-Take’s new physical-to-virtual offering
brennels Double-Take Move, I assume?
erson Ray, the SQL servers is the heavy workload by far, so we're working on getting some good measurements of the expected I/O
Ray_Weinstein Have you seen the latest Microsoft white paper on Hyper-V and SQL?
erson Yes
ranjith_purush Erson, you are very well positioned with the PowerEdge M710s
Ray_Weinstein It's pretty good
erson About a month old, yes?
Ray_Weinstein Yes, end of December
erson We have an EqualLogic PS6000e as storage for the VMs, btw
erson Matt, what problems have you encountered? Any tips when moving this kind of solution into production?
Matt_Woodings_HotSchedules Up to this point I haven't experienced any problems, but I have seen the solution save my butt
Ray_Weinstein Erson, use PlateSpin Reconn to assess the environment prior to moving it over. This way you can be predictive based on dynamic models
Matt_Woodings_HotSchedules We had two VMs fail on a cluster because of a bad DIMM, but they were migrated to the next node keeping them online
erson Ray, will take a note on that
Ray_Weinstein Nice safety net
erson I'm actually glad that we don't host our software for our customers. It sure is a challenge to host an application like HotSchedules does
brennels How many of the virtual machines do you have using failover clustering?
Matt_Woodings_HotSchedules About 20 machines are in a failover cluster between two clustered nodes
erson What type of memory configuration do you have to handle 20 VMs on a single PowerEdge 2950 (if the other node fails)?
Matt_Woodings_HotSchedules We have 32 GB memory in each one (the maximum at the time). However, currently the most cost-effective size today is 64 GB memory
Ray_Weinstein Erson, we also recommend using Datacenter edition to get the most out of VM guest licensing
erson Not with 11th-generation servers. We're a Microsoft partner so we have a couple of Datacenter edition licenses. 11th-generation with 18 memory slots is cheap with 4 GB for a total of 72 GB
Matt_Woodings_HotSchedules Each of the VMs is 32 bit. I am in the process of testing our environment in a 64-bit realm. So far with awesome results
erson We're going 64 bits on all fronts. Hell, 32 bit is dead on servers anyway. Windows Server 2008 R2 does only come in 64 bit
Ray_Weinstein Good point
brennels Matt, any tips or suggestions when looking for this type of Dell, Double-Take, and Microsoft Hyper-V solution?
Matt_Woodings_HotSchedules I can't speak to everyone, of course, but the MD3000i plus PowerEdge 2950III and Double-Take provides a very cost-effective solution to the SMB company
erson I'm sure Dell is eager to get you a couple of PowerEdge R710s soon enough ;)
Matt_Woodings_HotSchedules The PowerVault MD3000i in my experience is a great workhorse and coupled with Double-Take then provides replication
erson That CPU power is impressive coupled with 18 memory slots
Ray_Weinstein And some PowerEdge R900s :-)
erson And four NICs as well
KongY-Dell Just to remind the audience, the awesome case study on which this chat was based is located at: www.dell.com/content/topics/global.aspx/casestudies/en/us/us/fy2010_q1_id1194?c=us&cs=555&l=en&s=biz
erson As usual, there will be a transcript of this chat tomorrow (or on Thursday) on the Dell TechCenter. Everyone here from the U.S.?
KongY-Dell Yes, except for you...right, Erson ;)
Ray_Weinstein Austin, TX here
erson I actually wasn't the only Swede here last chat
Matt_Woodings_HotSchedules I am actually representing Queen and Country, but live in the U.S.
Brady_Lambert Houston here
Ray_Weinstein Tak (sp)
erson Almost, it's spelled tack :)
KongY-Dell Thank you folks for your participation
Ray_Weinstein LOL, I couldn't recall from my time spent in Stockholm
Brady_Lambert Thanks, see you next week
erson Anyway, great talking to you all. Very nice to hear from a company that actually has implemented something very similar that we're going for
brennels Good talking to you guys, see you next week
Ray_Weinstein Thanks all, cheers!
erson Next time it's VMware vSphere
brennels See ya, Erson
Matt_Woodings_HotSchedules It’s great talking to like-minded people with a need for uptime and DR
brennels Yea Baby...
erson Brennels, yes, I'll be here for sure
brennels Ta
ranjith_purush Thank you. Bye.
KongY-Dell Just as a reminder... transcripts of this chat will be posted tomorrow. Thank you kindly
KongY-Dell Thanks erson as always :)