This discussion focuses on Microsoft® Exchange 2007 software performance and scalability on the VMware® vSphere™ 4.0 virtualization platform. Featured speakers, including Todd Muirhead, former member of the Dell TechCenter team and known to the Twitter community as VirtualTodd, discuss how the vSphere 4.0 infrastructure can support different levels of Exchange users and achieve low latency levels in virtualized Exchange configurations. In particular, Todd discusses the results of his latest study before the results get published. Todd is currently a staff performance engineer at VMware.

Technical Community - Background Reading

Chat Transcript

Dell-KongY Hi there Virtualtodd!
VirtualTodd Hi everybody
Dell-KongY Hi there Jeffs aka Dell TechCenter's very own SANpenguin
Dell-KongY Welcome Martin!
grealish Hi Martin!
VirtualTodd Josh!
Dell-JeffS Hello
erson Can we expect any Windows 7 drivers from Dell before 22 October?
Dell-KongY So far I count two VMware staff engineers: the one and only Virtualtodd and Josh Schnee (jpschnee)
jpschnee Thanks. I thought I would listen in and see how today's chat went
grealish So where are we all logging in from?
erson Sweden!
eczerwin Chicago
martin Ireland
VirtualTodd Windows 7 and Windows Server 2008 R2 will be great guests on vSphere
grealish Switzerland
eczerwin Where in Chicago? I am moving there for work
Dell-KongY Welcome Marcdell and Dustin_lema!
MarcDELL Good afternoon
grealish @virtualtodd, I'm testing Windows 7 and Windows 2008 R2 on vSphere already
Dustin_Lema Thanks
Dell-KongY Welcome Chris_has!
grealish @eczerwin, Zurich area
eczerwin Hmmm…me too. @grealish, you aren't Swiss are you?
Chris_ Thanks and thanks for sending this out today
erson /me should get a quad-core play box to fool around with VMware
grealish @eczerwin almost ;)
erson Hi Scotth
DELL-ScottH Hello!
grealish Welcome Scott!
DELL-ScottH Wait, let me yell a little louder...Hello!!
Dell-KongY Welcome Scott H!
VirtualTodd Hey Scott
Dell-KongY Welcome Angel! Wow, I guess it's about time to begin
DELL-ScottH Hey Todd, what is up my friend? How was the Dell diamond on the 4th?
Dell-KongY Thanks everyone for joining the TechCenter and our distinguished guest, Todd Muirhead. Today, Todd will discuss the results of his soon-to-be published white paper on Microsoft Exchange 2007 on VMware vSphere
Dell-KongY Welcome Depping!
depping Hey guys
Dell-KongY We have quite a VMware contingent including Todd, Josh, and Duncan Epping
Dell-KongY Welcome Meza!
VirtualTodd It actually will be a blog post on Vroom!
Meza Hi Kongy
Meza I have some issues with IT Assistant, which room should I go to? Oh and this is my first time checking out the chat
DELL-ScottH Vroom! link for those that aren't looking for the vacuum cleaner: http://blogs.vmware.com/performance/
erson Meza, there is only one chat room
VirtualTodd Some quick background on Exchange and VMware
DELL-ScottH Meza, welcome to the chat; what is your issue with IT Assistant?
DELL-ScottH Chatting with Meza in private
Meza Okay, thanks Erson
VirtualTodd Kong and I presented a session at VMworld 2007 on Exchange 2007 on ESX 3.5
Meza Hi Scott
DELL-ScottH Hello
VirtualTodd At the time it was kind of edgy to talk about running Exchange on ESX
Meza ScottH, I currently have 2 problems: 1. I have two servers recently and they both have amber lights on. I looked at the LCD, and it indicated some sort of power problem. Looked into the Server Administrator, which reveals that both power supplies are said to have predictive failure
VirtualTodd Our result showed that performance of Exchange on ESX was good; with latencies well below the standard set by Microsoft for good performance
Dell-KongY www.delltechcenter.com/page/exchange+and+vmware and www.dell.com/downloads/global/solutions/virtexchange2k7_finalfrontier_vmworld2007.pdf
Meza ScottH, the problem is, in our monitoring the server still shows a green check like nothing is wrong with it
Dell-KongY Welcome Bm7150, Veeammark, and Ron!
Meza ScottH, we only found the problem because someone was walking in the data center and saw amber lights
VeeamMark Thanks @dell-kongy
Meza ScottH, can we get IT Assistant to send out notifications on these?
VirtualTodd That portion of the testing was just straight Loadgen testing that measured latency or response time across a range of actions done by Exchange users
DELL-ScottH Meza, is the SNMP trap destination in OMSA pointing to the IT Assistant box? And, yes, but first you have to make sure that OMSA is set correctly to forward the traps
VirtualTodd In addition, we tested and measured the performance of Exchange during vMotion and HA failover
Meza ScottH, yes, if we pull the power plug to simulate a power failure, it will trigger a red X and send out a notification
VirtualTodd Again, we found good performance—with only limited hits to performance during vMotion
Dell-KongY Welcome Durkinr!
VirtualTodd And reasonable recovery time for HA failovers
Dell-KongY Welcome Delldustin!
DELL-ScottH Meza, okay, so traps get sent to IT Assistant correctly. On OMSA have you filtered out lower-level events? Let me pull up one of my boxes—what version, which servers?
Meza ScottH, okay, let me take a look to report to you the version we are running
VirtualTodd The testing that I did recently focused on using Exchange in a DRS cluster (with vMotion being used to auto load balance)
Dell-KongY Welcome Nakul1 and Dennis_s!
VirtualTodd One of the big differences between the earlier tests and the recent test is the size of the workload. In the earlier tests we tested up to 4000 users, but in my DRS testing I used 16,000
Dell-KongY Welcome Cheddr and Sdeuskar!
VirtualTodd So it was with a fairly large number of users
Meza ScottH, we are running Windows Server 2008 standard 32 bit with IT Assistant version 8.3.0
grealish @virtualtodd, any tests with DPM and comparing power saved over without VMs?
Meza ScottH, and the server is joined to the domain for authentication
VirtualTodd Anyway, the results showed that DRS was able to manage the 8 Exchange VMs in the test such that performance improved versus if DRS was not used
Dell-KongY So a key takeaway is to allow DRS to manage your VMs’ resource needs
VirtualTodd I haven't done a DPM testing to combine with the DRS tests, but knowing that using DRS does not negatively affect Exchange means that it should work very well
Dell-KongY @virtualtodd, what was the profile of your Exchange workload?
DELL-ScottH Meza, on the servers with OMSA, under the Alert Management tab, SNMP Traps, are all the boxes checked for "Information, Warning, Critical"?
VirtualTodd To get a workload that would change a bit over the course of the test I modeled a company that was spread across the four time zones of the U.S.
Meza ScottH, okay, let me check that
VirtualTodd This company just so happened to be exactly evenly spread across the four time zones with 4000 users in East, Central, Mountain, and Pacific
DELL-ScottH Meza, was the predictive failure a "non-critical" alert? Or what severity did it report?
VirtualTodd So at the beginning of the test 4000 users start, and then an hour later 4000 more, etc. Toward the end of the test the first 4000 users quit for the day first, then the next 4000, etc.
Meza ScottH, yes, it's all checked informational, warning, critical
VirtualTodd So at the beginning and the end there are opportunities for DRS to rebalance the workload. Hardware details: Dell PowerEdge M905 blades and Dell EqualLogic storage
Dell-KongY So how many of you have deployed Exchange VMs on vSphere? Or are you waiting for reference solutions/architectures?
grealish Not yet, but planning along with a few other bits
Meza ScottH, everything has a green check. I had to drill down from System > Main System Chassis to Power Supplies to see the yellow triangles
VirtualTodd I guess that's not really details, more like very high-level description :)
eczerwin We did but then we never moved to it—not in production
grealish But good that you mention PowerEdge M905 blades; we had a design spec done to run vSphere on
DELL-ScottH Meza, okay, under Alert Management, Platform Events is the "enable Platform Event Filter alerts" box checked?
VirtualTodd Here is a white paper that details performance of Exchange on vSphere that was done by one of my colleagues on the VMware performance team: http://blogs.vmware.com/performance/2009/05/exchange-2007-performance-on-vsphere-4.html
DELL-ScottH Meza, hmmm, the yellow triangles didn't bubble up?
VirtualTodd I have talked to many customers that run Exchange on VMware
Dell-KongY @eczerwin, if you don't mind sharing, what's keeping you from virtualizing Exchange in production?
Meza ScottH, no, the higher levels all have green checks
DELL-ScottH Meza, which server model?
Meza ScottH, Enable Platform Even Filter Alerts box is not checked
Dell-KongY How many of you are waiting for Exchange 2010 before you virtualize Exchange?
VirtualTodd Perhaps one of the most interesting case studies I have seen is the one that details VMware's deployment of Exchange internally (on ESX 3.5): www.vmware.com/files/pdf/solutions/08q4_vm_exchange_server_2007_vi3_wp.pdf
Meza ScottH, both servers are PowerEdge 6800
DELL-ScottH Meza, at a minimum, that alert should be showing on the Health page of the main "system" page, is it not?
Meza ScottH, nope, System Page is showing two green checks
VirtualTodd I'm really impressed that nobody has clicked a link and accidentally left the chat!
Dell-KongY Thanks for sharing those links, Virtualtodd
Meza ScottH, Main System Chassis, check, and Storage, check
DELL-ScottH Meza, sounds like something is broken in the firmware; latest is on the server?
Meza ScottH, which firmware? BIOS? BMC?
erson Virtualtodd, maybe we all are using IE? a buggabuggabooo :)
eczerwin @dell-kongy, nothing against the solution; the tests we did were just fine...there was just a change of plans, and we decided to consolidate data centers at a later time
DELL-ScottH ScottH, links have been working better in IE 8 :-)
Dell-KongY @eczerwin, understood
erson I can click links with either right- or left-click using IE 7 with no problem whatsoever
VirtualTodd Some people used to have issues or fears about running Exchange on VMware, but I think that those concerns have gone away almost completely. There are lots of performance papers and case studies showing that the performance is good
Meza ScottH, Remote Access Controller firmware version 1.60, Baseboard Management Controller version 1.77, Peripheral bay backplane version 1.00, Backplane firmware version 1.00
VirtualTodd It is also now an officially supported solution. Anybody run into issues or roadblocks lately with running Exchange on ESX?
Meza ScottH, side note: does IT Assistant not work well with Windows 2008?
DELL-ScottH Meza, there's DRAC 1.75 and BMC 1.83 available on the support site. I'd start there; seems like a bug, but I'm not in support so not sure if it's a known thing. IT Assistant works fine for me in Windows 2008
Dell-KongY Just curious, how many Exchange users do you guys build your VMs out for? Todd's study has shown 16k users defined, tested, and supported. Also, do you prefer more VMs with fewer users or fewer VMs with more users supported?
DELL-ScottH Meza, and there's A06 system BIOS
Dell-KongY Personally, I like more VMs with smaller numbers of users to take advantage of vSphere features
grealish One vCPU VMs with FT ;)
Dell-KongY @grealish, agreed
grealish And faster vMotion in DRS setups
Meza ScottH, okay, I'll keep that in mind, since it's a heavy database server they are not going to let me flash firmware without good reason. I'll do some research
VirtualTodd There has been some work done with Exchange VMs and FT: https://powerlink.emc.com/nsepn/webapps/btg548664833igtcuup4826/km/live1/en_us/offering_technical/white_paper/h6260-virtual-infrastructure-clariion-cx4-120-vsphere-wp.pdf
Dell-KongY Anyone using Exchange's HA/DR features over VMware's HA/FT/DR features?
VirtualTodd Have to have a powerlink account
grealish I've noticed that in our tests, vCenter really works better with smaller VMs; we could get more performance that way
Meza ScottH, is version 8.3 the latest?
DELL-ScottH Meza, yeah, I hear you on that. It seems like a bug; you can call into support and see what they say, but all the errors should bubble to the top health screen
VirtualTodd I have also seen that 2vCPU VMs will perform better than 4vCPU VMs
Meza ScottH, gotcha
DELL-ScottH Meza, there is a version about to release as part of the OpenManage 6.1 release, in a few
VirtualTodd Although the difference in performance might be imperceptible to most users
DELL-ScottH Meza, 8.3 is currently the latest; let me pull up my release schedule
VirtualTodd In general, you have more flexibility with smaller VMs to load balance among hosts
Meza ScottH, ae the Software Updates not working anymore on IT Assistant? I can't get my Online Repository to sync
VirtualTodd Is anybody considering using FT for Exchange? Or another workload?
Dell-KongY So @ virtualtodd, what are your next focus areas as far as vSphere performance testing?
grealish Yeah, it would make sense for load balancing to work better with smaller slot sizes
Meza ScottH, lastly, do you know how I can enable scripting permanently on the server? I know that IT Assistant relies on VBS to send out e-mail
DELL-ScottH Meza, they may have hosed up the catalog file again...arrggh, seems to happen near a release
VirtualTodd I'm actually working on some Exchange FT testing right now; thanks to all in the Dell TechCenter lab
Meza ScottH, yeah, I was trying to update it to check all my servers to see which ones are out of date, but can't ever get it to sync...I thought maybe it's because I moved it to Windows 2008
VirtualTodd The stuff after that is actually under discussion right now, but looking at some multi-workload testing like maybe Exchange and SQL on a vSphere internal cloud
DELL-ScottH Meza, I have another guy that pinged me about Content Manager not synching and uses the same catalog, so I'm sure something got corrupt on our side—has happened before. Please call that in so we can get more customers logged so they will work on it faster :-) I'm not sure about your question on permanently enabling scripting
Dell-KongY Very cool. Heterogeneous workload testing in virtualized environments is an awesome topic
Meza ScottH, this happened to me before in Windows 2003, but one of the IT Assistant gurus got it fixed for me. Basically, when sending out notifications IT Assistant will trigger a VB Script to send out the alert. On my server, I think it's a GPO or something; I'm not sure that disabled any scripting
Dell-KongY There is always that question of what applications should run on the same system to be the most efficient
Dell-d_glynn Todd, it sounds like you'll have some great examples for virtualizing the last 5 percent of the data center
grealish We have a political problem with customers understanding private clouds in heterogeneous setups
Meza ScottH, I have a script enabler that I can run when I log it and then notifications will work again, but as soon as I log off the scripting is disabled again, so no more notifications
grealish And that I'm really interesting in hearing about tests
DELL-ScottH Meza, ah, so a Windows security thing that is not allowing execution of the scripts
Meza ScottH, yes!
VirtualTodd I agree that most of what we are looking at now is that last 5 to 10 percent
DELL-ScottH Meza, hmmm, is it just an executable? I think there are tools that can turn it into a service
Meza ScottH, I wish I remember what the other guy did to get that to work, but I forgot. It's been a few years
VirtualTodd grealish, can I quote you on that?
Meza ScottH, well, about a year, not a few years
DELL-ScottH Meza, that would be a hack though
Dell-KongY In that last 5 to 10 percent, we are also looking at HPC workloads :) In virtualized environments
grealish Sure
VirtualTodd In theory, thinking of things as a cloud should be easier
Meza ScottH, oh I see, I thought you might have a more elegant solution, but yeah I guess I can make it run as a service
VirtualTodd I believe that management of a cloud will be different in some ways than what we are familiar with
Meza ScottH, it's called noscript.exe
DELL-ScottH Meza, well, if I were a better Windows guru, I might know how to fix it right :-)
grealish Yes, but for people this is really a new IT philosophy that they find hard to grasp
DELL-ScottH Meza, that was just my quick, off-the-cuff, "well this might work"
VirtualTodd Philosophy and politics: I agree that those things are hard
Dell-d_glynn HPC in the cloud? Very interesting
Dell-KongY @grealish, is it more about ownership and utilization of resources?
grealish Like I have a management meeting tomorrow, and I hope they won't bring this up :(
Meza ScottH, ha ha, I'm sure you are a guru, but you just don't know about this one. So you haven't heard of anything like this with IT Assistant huh?
grealish Like in firewall when management sees access policy, user groups, they like these words :)
DELL-ScottH Meza, the more I learn, the more I learn I need to learn :-) I'm digging through the documents now to see if they have mention of it: http://support.dell.com/support/edocs/software/smsom/6.1/index.htm
grealish And it makes it very easy to sell this
Meza ScottH, BTW are they killing IT Assistant?
DELL-ScottH Meza, 2. Windows Scripting Host (WSH) has disabled the running of VB scripts. By default, WSH is installed during the OS installation. WSH can be configured to prevent scripts with a .vbs extension from being run. On the desktop, right-click My Computer and then go to Open > Tools > Folder Options > File Types and look for the extension .vbs and verify that File Types is set to "vbscript Script File"; if not, click Change and choose "Microsoft Windows-Based Script Host" as the application that runs the script. And, yes, IT Assistant is being replaced by the Dell Management Console
VirtualTodd But clouds are more difficult to grasp?
DELL-ScottH Meza, still have a little more than a year
VirtualTodd (For management)
Meza ScottH, is that OpenManage 6.1?
grealish For me, no; but anything new first approach is
DELL-ScottH Meza, yes, but should be applicable for 6.0.1 too. From the readme: http://support.dell.com/support/edocs/software/smsom/6.1/en/readme/smconsole/readme.txt
grealish However, will be interesting to see Exchange tested in heterogeneous setups, like with SAP. Most companies have this
Meza ScottH, awesome with the WSH article...see, you are now a higher guru. But looks like the write up is for 2003. Hmmm...wonder what that is under 2008... I'm looking
VirtualTodd I actually have been learning a little bit about SAP in the last few weeks; so that would be a good match for my testing
DELL-ScottH Meza, oh how I love that they change the location of everything...keeps us all employed
Meza ScottH, yeah, they have to confuse us so we get recertified
DELL-ScottH Meza, and 6.1 was released on 7/2: http://support.dell.com/support/downloads/download.aspx?c=us&l=en&s=gen&releaseid=r226596&systemid=pwe_6800&servicetag=&os=wnet&osl=en&deviceid=554&devlib=0&typecnt=0&vercnt=11&catid=-1&impid=-1&formatcnt=1&libid=36&fileid=324791
grealish These big names are for sure the selling test!
DELL-ScottH Meza, we are all certifiable!
Meza ScottH, oh, that's a good one!
Dell-KongY @grealish, so I take it that an Exchange + SAP heterogeneous study would have an audience? ;)
grealish But can I ask, how do you generate all the user workloads in your tests? @dell-kongy, yeah for sure!
VirtualTodd For SAP, their SD benchmark is what I would use (that's what I learned how to set up and run). Exchange is the Loadgen tool from Microsoft
Meza ScottH, no way! In 2008 changing of File Types is moved to Group Policy Management Console?!
grealish You look at any large organization, they all have purchasing and finance departments, and the two heavy applications are e-mail and SAP. Okay, Citrix too for remote ;) Maybe BlackBerry connector too for the CRAIC ;)
Dell-KongY My, oh my. How time passes by when you're having fun. Before I forget, thank you to Mr. Todd Muirhead!
VirtualTodd Thanks Kong for inviting me for today
Dell-KongY And thanks to the audience for joining us for today's chat!
VirtualTodd Thanks to everybody for the great discussion and questions
grealish NP guys, interesting chat
Dell-KongY As always, the transcript of today's chat will be available tomorrow for your reading pleasure
erson The much-improved I/O in Exchange 2010 will push people even more into virtualizing Exchange. There isn't much left that's hard to virtualize as long as you have suitable hardware
DELL-ScottH What'd I miss? Was having great side chat session with Meza
Dell-d_glynn You can catch up in tomorrow transcript :p
grealish Sorry Scotth, world blew up and imploded again ;)
DELL-ScottH Dang! Again?! Where are the transformers when you need them?
erson Hopefully not in a third movie
DELL-ScottH Thanks d_glynn :-) Fourth, 5th, and 6th!
Dell-d_glynn Any time, Scott