With the recent release of the VMware® vSphere™ 4 platform, this discussion picks up from a previous TechTuesday chat (June 2, 2009) to focus on performance to achieve the benefits of cloud computing and utilize other platform features.

Featured guests include:

  • Todd Muirhead, VMware staff engineer, performance
  • Jeffrey Buell, VMware staff engineer, performance
  • Josh Donelson, VMware staff engineer, Dell | VMware Alliance
  • Dushyant Dwibedy, Dell systems engineer, virtualization solutions engineering

David J. Morse is a server performance benchmark lead, Office of the CTO, at Dell and has been a performance engineer on Dell’s Server Performance Analysis team for 10 years. David has participated in multiple industry standards bodies, including the Standard Performance Evaluation Corporation (SPEC), where he is currently chair of the Open Systems Steering Committee (OSSC). His current focus is on SPEC, Transaction Processing Performance Council (TPC), and VMware benchmarks. David has a B.S.C.E. from the University of South Carolina Honors College and is a Red Hat Certified Engineer (RHCE).


Technical Community - Background Reading


Chat Transcript

Grant Are the Dell guys here?
grealish Almost ;)
grealish We are a bit early
Grant About an hour or so?
grealish Okay I'm here
Dell-KongY Welcome Bananabob, Grant, grealish, and Joshwillis
grealish Does this chat start in 30 minutes?
Dell-KongY Yes Grealish, the chat will start in about 40 minutes
Dell-KongY Welcome
grealish Cool thanks, about 9.25 pm here!
HoosierCAB A little early, but Twitter told me to be here or be square :)
grealish I see the Dell guys are here; have we some VMware pros yet?
HoosierCAB Grealish, I'm far from a pro, but I deal with VMware on a daily basis
Dell-KongY Welcome markmathson
markmathson Thanks, glad to be here
DELL-ScottH Welcome everyone. Be nice to Kong and Jeff...I have customer meeting in four minutes, so will not be back for a few
Grimsrud Thanks, Kongy
markmathson I will be on and off as well; have a conjoining teleconference call
Dell-KongY Waiting on three VMware staff engineers
Dell-KongY Welcome Josh. Josh is a VMware staff engineer
jdonelson Hello Kong. Hello all
ceri Before we start, I assume the transcript will be made available as usual?
VirtualTodd Am I in the right room?
Dell-KongY Welcome Virtualtodd. VMware staff engineer focused on performance
Dell-KongY Welcome Techdad
Dell-KongY @ceri, yes, the transcript will be made available as usual
VeeamMark Howdy all!
ceri Great, thanks Dell-kongy
Dell-KongY Welcome Veeammark, Vmdoug, and claystorm
VMDoug I made it. I was stuck in the lobby for a minute
jdonelson Mark, you're everywhere!
Dell-KongY We have Veeam representation :)
Dell-KongY And by their two finest engineers no less
VMDoug Of course ;-)
VirtualTodd We are joined by greatness—Vmdoug is here!
VeeamMark Josh, yes I am
VMDoug Oh, I'm not that great
Dell-KongY Welcome David J. Morse, team lead of Dell's performance team
VeeamMark That's not what Cheryl said, Doug
DELL-ScottH @virtualtodd, errors on blade were ECC; looks like one time, must have been alpha particles—I cleared it. Also, about to do EqualLogic demo with customers so taking some snaps and creating LUNs on the SATA
grealish Hey Veeam, hear hear!
VeeamMark Scott, Network lice can also be blamed occasionally
VirtualTodd I was trying to figure out how to clear—how did you clear it?
VeeamMark Howdy Grealish
grealish Cool competition, but couldn't get my video in time :(
Dell-JeffS Taking naps?
VMDoug We just met with the Ohio EqualLogic reps—great guys
Dell-KongY We'll wait a few more minutes and then we'll begin
VMDoug @grealish, sorry, we tried to give plenty of time but I guess like me you always wait till the last minute ::blink
VeeamMark Grealish, did you vote for the finalist though?
JasonPowell /me waves
grealish @veeammark, are the vids live now?
VeeamMark @vmdoug said I was not eligible to win
Dell-KongY Welcome Jasonpowell
VeeamMark Yes sir, if you are on Twitter, check out @dell_veeam for the link
VirtualTodd Hey Jason!
claystorm Only problem is Twitter is going down
grealish Cool thanks
DELL-ScottH Check my tweet just now to see Jeff hard at work this very second! :-)
claystorm Never mind, forgot about the time difference
dave_graham Hey all
Dell-KongY Welcome Dave
grealish Okay all, going to start with a question
dave_graham Hey Kong
VMDoug Howdy Dave
dave_graham Long time, no talk
grealish I've been evaluating Dell blades plus EqualLogic SAN and vSphere 4 for the last four weeks
dave_graham ;)
Dell-KongY For your reading pleasure, “Microsoft Exchange Server 2007 Performance on VMware vSphere 4: http://www.vmware.com/resources/techresources/10021; Virtualizing Performance – Critical Database Applications in VMware vSphere: http://www.vmware.com/resources/techresources/10013; Virtualized SAP Performance with VMware vSphere 4: http://www.vmware.com/resources/techresources/10026; Performance Evaluation of Intel EPT Hardware Assist: http://www.vmware.com/resources/techresources/10006; Performance Evaluation of AMD RVI Hardware Assist: http://www.vmware.com/resources/techresources/1079
Dell-KongY Just some of the great reference material that VMware has produced
VMDoug Remember, right-click those links
Dell-KongY Thanks Vmdoug
grealish And we've successfully tested our setup on it; however, we have questions regarding performance and the new v7 VM settings
VeeamMark Fire away, Grealish
ceri Grealish, which blades?
Dell-KongY Welcome Billglick
Dell-KongY Let's get started
grealish PowerEdge M605, 2x AMD 2300 32 GB RAM
VeeamMark @virtualtodd, How many mailboxes did you get on a full-height blade?
Dell-KongY Welcome everyone to the TechTuesday Web chat. Today's talk is around VMware vSphere 4 performance
grealish EqualLogic SAN iSCSI PS5000
Dell-KongY We have performance engineers from both Dell and VMware to answer your questions
grealish The hardware is okay; well, have a question regarding the Broadcom 5709s about TCP offload support in ESX 4
BillGlick Is this open discussion of topics/questions?
Dell-KongY Plus, we have many "virtual" friends from the industry who have also joined
VeeamMark @billglick. absolutely...fire away with questions
Grant How many XP VMs should we expect to run on one server with 128 GB RAM dual RAID cards with five drives each?
Dell-KongY @billglick, yes this is an open discussion on vSphere performance on Dell
Dell-KongY @grant, it depends :)
VeeamMark Grant, what is your normal XP image configuration?
Grant Running vSphere 4
Grant Yes
Dell-KongY What's your VM configuration? What's your workload profile, etc.? Is it self-similar work that can take advantage of memory sharing?
grealish @grand, I guess a lot of disk I/O is going to be a limiting factor there on a single host with local disks
Grant Still studying the profiles; general office needs
jdonelson @grant, in general, XP isn't all that disk intensive
BillGlick Looking to move to vSphere for data-intense services like MYSQL and Zimbra, both on Red Hat Linux. How can I start to spec servers and SAN for an I/O-intense sphere setup?
jdonelson Our sizing rule of thumb for the desktop is four to six (perhaps slightly more) per core on the server, provided you have the memory and raw disk space available
bananabob For that kind of sizing, what are the memory and raw disk specs?
Dell-KongY @billglick, do you already have your physical performance metrics for disk I/O?
VeeamMark The key in virtualizing XP or enterprise loads is always to figure out what the current physical performance metrics are so you can have a tangible figure to measure against
BillGlick I could get them, but don't have them handy. Just trying to figure out how to even work with Dell on specifying hardware
jdonelson I agree with Mark; starting from actual concrete data turns guesswork into planning
VeeamMark Bill, if you have the metrics available, most of Dell's ASG and GICS personnel are trained in scaling and architecture
ceri I've seen white papers, but what's the practical performance drop-off with putting three DIMMs on a channel for Nehalem? 48 GB as a "practical" maximum is making we wonder whether I should stick with Penryn
Dell-KongY @billglick, we have a virtualization advisor tool: http://advisors.dell.com/advisorweb/advisor.aspx?advisor=c82c3ec8-c94f-4602-9a41-c20382db1cd0&c=us&l=en&cs=555
Grant Is the virtual router included in the vSphere 4 Essentials Plus bundle?
Dell-KongY Welcome, Drez
Dell-KongY Welcome Brett
Brett Howdy
BillGlick What is an ASG or GIC?
drez Greetings
Brett I'll listen in for the moment. Hi all
grealish VM question—I virtulized an application server we have that ran on a dual quad core, (eight cores) with 4 GB RAM; however, in our tests there was higher rate of Interrupt calls than usual; the application does a lot of open/close of log files
VMDoug @grant, you mean the vSphere-distributed switch?
Dell-KongY @billglick, Dell’s Advanced Systems Group and Global Infrastructure Consulting
VeeamMark Bill, ASG is Dell's Advanced Systems Group, also known as technical sales reps or systems consultants; GICS is Dell's service arm, called Global Infrastructure Consulting Services
Grant I need to rout ports from several VMs to one IP
Dell-KongY Welcome Dushyant
Dell_Dushyant Hey Kong
BillGlick So, our sales rep could connect us with someone from ASG or GIC?
DELL-DavidJMorse Ceri, we have not seen an appreciable drop in vSphere performance going from 2 DPC to 3 DPC
VeeamMark Bill, yes
DELL-DavidJMorse It really depends on how memory bandwidth-intensive the applications are though (YMMV)
Guest_16 Afternoon folks!
Dell-KongY Welcome Guest_16 and plauterbach
Dell-KongY @grealish, is your question around the higher rate of interrupt calls?
VirtualTodd Grealish, was there also a drop in performance with the increased rate of interrupt calls?
VeeamMark Bill, I work with both ASG and GICS on monitoring and measuring virtual machine performance, so feel free to contact me as well if you need help
grealish Yes, big drop
VMDoug @grant, I think the v'
ceri Dell-davidjmorse. I think "not very" is the answer in most of our workloads' cases. Good to know that you've not seen too much drop off. Do you know if 3 DPC bandwidth on Nehalem is still better than Penryn? I suspect yes, but don't know
grealish But it's partly a poor program, we have a flat memory database
VMDoug @grant, I think the vnetwork Distributed Switch is only in Enterprise+, not Essentials+
Guest_16 You are correct. Vmdoug
ceri @vmdoug, that's correct
VMDoug (Right-click) http://www.vmware.com/files/pdf/vsphere_pricing.pdf
grealish And is also I/O intensive; however, tomorrow we have a management meeting and I'd like to get some information regarding how vSphere ESX deals with interrupt calls
Brett In reference to Grant's question, could a Dell switch handle that outside of the VM environment?
Guest_16 My general testing and deployment (using FreeBSD for my guests) is showing higher system time (vmstat, systat, and top) than running the same VM on 3.5 U4 These are mail and Web servers
grealish @dell-kongy, btw, it's Windows 2003 32 bit
DELL-DavidJMorse Ceri, checking
HoosierCAB @grant, I'm confused: "route ports from several VMs to one IP"?
Guest_16 I think the route ports from several VMs to one IP is perhaps a router/firewall doing Nat to the outside world? Could be wrong...
HoosierCAB @grant, that usually implies load balancer or layer 3/4 playing
Grant I only need access to one port on each VM
grealish @grant, @hoosiercab, sounds like you want to put a view VMs behind a Nat
VirtualTodd Grealish, are there other VMs running at the same time?
grealish And use port forwarding
HoosierCAB @grant, if that's the case, m0n0wall's VM appliance works like a charm
VirtualTodd If CPUs are overcommitted you will see a high interrupt rate in esxtop as switching between VMs occurs
grealish @virtualtodd, yes, about 26 on a cluster with 2x PowerEdge M605 (16 core, 64 GB + 700 GB 15K iSCSI SAN)—really overkill
VeeamMark Does Dell still publish storage best practice documents for Exchange?
Dell-KongY Welcome Vmware-jeff
dave_graham Whew. What did I miss?
VeeamMark @dave. I think that Scott called your name for a free Dell DJ
VMDoug Anyone notice a slow down in throughput on the service console with ESX 4?
VirtualTodd Hey Jeff!
VMDoug (Right-click) http://communities.vmware.com/thread/213225
grealish @wmdoug, a bit with enabling SNMP on the ESX host
dave_graham Lol @veeammark, about bloody time
ceri @dell-davidjmorse, thanks, appreciated
VirtualTodd Grealish, if the application is particularly sensitive to CPU cache, then the switching can cause a higher cache miss rate
Guest_16 So, am I the only Unix-like person here seeing higher system time in use on their VMs on 4.0 versus 3.5 U4?
VeeamMark Guest_16, you are not alone
grealish Guest_16, I could say I find Cacti to run slower on v4 than 3.5; its running on a Ubuntu VM, but haven't really had time to look into it
dave_graham @guest16, I can test that out for you if you want
DELL-DavidJMorse @ceri, the answer is, yes. The memory bandwidth increase is still huge from Penryn/Harpertown (< 8 GB/sec. with Stream) versus 27 GB/sec. @ 3 DPC w/ Nehalem
Guest_16 Cacti will only report items out of SNMP; this is directly on the servers (I have > 200 servers I manage), and the ones I am testing on 4.0 are showing higher system time than on 3.5 U4 (as I said). Cacti won't help me :(
Guest_16 Hmmm...but your point of Cacti running slower might have a hint for me
ceri @dell-davidjmorse. thank you so much; that's extremely useful
DELL-DavidJMorse Sure thing
Grant Any tips on running XP; what should I uninstall or disable?
grealish Guest_16, did you port the VM from 3.5 U4 to v4, and did you update to VM v7? Also, the VMI can radically improve things
Guest_16 My VMs are FreeBSD, and you are seeing something on Ubuntu. I could try to do a tweedle-beetle-battle
Guest_16 Yep, took my templated system, deployed to older 3.5 U4 system and to 4.0 system, edited up configurations, and then put them into production passing mail or Web traffic (depending on version)
VeeamMark @grant, ignore the perf mon statistics in VMs
grealish Guest_16, do a clean install and investigate your VM configuration; could be some modification that may improve things a little
Guest_16 Marked increase in system time, anywhere from 3–15 percent
grealish I'm noticing the same decrease in CPU performance from a P2V setup
Guest_16 That'd be pretty easy to do as I didn't customize the VMX files at all. They are already pretty generic, but that might be fun to try
grealish Okay, getting greater than that, and as @virtautodd mentioned, could be a higher miss rate
Grant @veeammark, why ignore?
VirtualTodd Guest_16, on what hardware did you see the increase in system time?
VeeamMark @grant, within VMware, guest OS perfmon statistics are skewed
Dell-KongY @guest_16, did any other performance metrics increase as well?
VeeamMark @grant, VMware has a special technology built in for performance, memory ballooning, taxing, and CPU affinity
Dell-KongY @grealish, what P2V tool did you use?
grealish Vconverter
Dell_Dushyant @guest_16, if using a second HDD, I would try using the direct path I/O driver as well
Guest_16 Hardware can be PowerEdge 2900III or PowerEdge R905
grealish One of the other guys just gave me a disk of workstation VMs
Guest_16 What other performance metrics do I need? This is production running (everything we do is load balanced/clustered), so I am seeing real-world items. I am not benchmarking. If you tell me what else you are looking for, I can sure see about providing it
markmathson Anyone have any resources to justify an upgrade, sooner instead of later, from V3 to V4? Particularly for environments with Exchange and Microsoft SQL?
Guest_16 For both PowerEdge 2900III and PowerEdge R905 systems, they are on the same SAN, using the same LUNs. Disk I/O looks about the same between the systems (using iostat, vmstat, systat)
dave_graham @mark, yeah, PP/VE ;)
Guest_16 No issue if no one has ideas; playing with a .0 release is always scary if you ask me. But fun! So I *must* continue to play...
VirtualTodd Guest_16. I would recommend using esxtop as well for measuring performance
markmathson esxtop has proven useful for me as well at times needed
ceri Any recommendations for 10GbE implementations? Don't need it sustained, but need to burst for backup
Guest_16 @virtualtodd, what will that tell me in terms of what I see inside the guest? I am not trying to be obstinate and more than willing to abuse another tool if it helps me figure out what I am seeing
dave_graham @ceri, use NFS for 10GbE
VirtualTodd FYI, here is a link to a guide to esxtop: http://communities.vmware.com/docs/doc-9279
VirtualTodd The best thing about using esxtop is that you can see all VMs, so it is a different view
vmware_-_jeff @guest_16, is your actual performance worse, throughput or latency? It's possible that time accounting could have changed in 4.0 versus 3.5 without any change in actual performance. That's why we recommend esxtop to measure utilization
ceri @dave_graham, why? Better use of the bandwidth? We have a well-established FC infrastructure, so it would be easier to stick with that
VirtualTodd Tools inside the VM or measuring performance from end-use point of view are also valuable
dave_graham @ceri, you mentioned 10GbE...so, how are you going to do FC over that? ;) Unless you're talking FCoE, and that's a completely different story
grealish I've been using process explorer in Windows to measure, but like piro—neater
Guest_16 Yes, I am seeing slower response on HTTP requests and SMTP command processing, which is why I am looking inside the VM and what I find is higher system time. I'll do up the esxtop to see what I get from it as well and see what it gives me
ceri @dave_graham, I meant for traditional network traffic ;-) Was going to do FC over glass like usual
HoosierCAB Someone tried to private chat me earlier—pop-up blocker killed it; feel free to now
Dell-KongY Welcome Drewroman
Guest_16 The URL given by Virtualtodd ends in Doc-9279—notice the case
ceri @dave_graham, what I wanted was, to each blade, dual port FC, 10GbE + 2x 1GbE
dave_graham Ceri, so, what's your? Again? Recommendations for FC under ESX 4?
ceri @dave_graham, just any tuning recommendations for 10GbE under vSphere. General implementation question, 10GbE on ESX is new to me. Any gotchas, etc. to look out for?
dave_graham Ah, so, implementation-wise, would highly recommend taking a look at Sakac's blogs on NFS and iSCSI. That will cover most of what you can do with GIGE and 10GbE
Dell-KongY Welcome Jgood
ceri @dave_graham, Great, thanks. blogs.emc.com?
vmware_-_jeff Doc-9279 works for me
dave_graham @ceri, just a second and I'll post the links
dave_graham NFS on VMware: http://virtualgeek.typepad.com/virtual_geek/2009/06/a-multivendor-post-to-help-our-mutual-nfs-customers-using-vmware.html
Dell-KongY http://communities.vmware.com/docs/doc-9279
ceri @dave_graham, thanks very much
dave_graham Can't seem to find the iSCSI one right now. Anyhow, also check out: http://blog.scottlowe.org/2008/12/19/vmware-esx-networking-articles/
Dell-KongY @dave_graham, thanks for the links
Grant Should I use the SCSI or the IDE driver? What are the pros and cons?
ceri @dave_graham, that's cool, thanks
VirtualTodd We have recently published a lot of new performance information on Vroom! http://blogs.vmware.com/performance/
dave_graham @grant ESX? Or workstation?
VirtualTodd Topics have included SAP, Java, Specweb, and Exchange so far. All on vSphere
Grant ESX 4
dave_graham SCSI
ceri With regard to pushing 10Gbe, 2x 1GbE and 2x 4/8GB/s FC to each blade, what blade/enclosure options should I be looking at from Dell?
VirtualTodd I have a paper in the pipeline on Exchange with FC, iSCSI, and NFS
dave_graham @ceri, PowerEdge M1000e with PowerEdge M805 or PowerEdge M905 blades ;) (I'm a little AMD biased.) ;)
Dell-KongY @ceri: http://www.delltechcenter.com/page/dell+virtualization+engineering+white+papers
ceri @dave_graham, @dell-kongy, thanks. ;-)
Dell-KongY Two white papers: “Business Ready Configuration” and “Networking Guide with Dell Blade Servers”
Dell-KongY They will be updated to vSphere shortly
ceri @virtualtodd, there is an EMC white paper on Exchange with vSphere and iSCSI
VirtualTodd Ceri, I have seen that paper as well—do you have the linky handy?
VeeamMark Help us send someone to VMworld free: http://www.veeam.com/forums/viewtopic.php?f=12&t=1050&start=0
VeeamMark Oops...sorry, wrong window guys! Cut and paste fail
ceri @virtualtodd, Yeah, it's…oops, can't cut and paste on the Mac. Just a sec…
ceri http://tinyurl.com/emc-pdf
Dell-KongY So as we are wrapping up, I'd like to thank our guests as well as all the attendees
VirtualTodd Thanks for hosting the chat, Kong
Dell-KongY Thanks to our friends from the virtual community for dropping by and sharing your expertise as well your knowledge and expertise are most welcome
ceri Thanks again to all for their help
grealish Thanks guys
jdonelson Thanks for hosting, Kong
grealish Any room for one more question?
dave_graham Shoot, Grealish
Grant Thank you
grealish Dave_graham, anyone get hardware adapter to work for iSCSI initiators in ESX 4?
grealish As in TCP offload support, hardware support for iSCSI traffic, etc.
dave_graham QLogic 4602cs should work
grealish And Broadcom adapters?
dave_graham General rule (from where I sit) Broadcom blows ;)
ceri @dave_graham, Hear, hear. Dell kit is still all shipping with it?
dave_graham Honestly, check the HCL and see if they're on there. If you need iSCSI TOE, QLogic has the best option
grealish Just to note, I've checked the ESX HCL, and it says Bcm5709s supports iSCSI, however doesn't list in the configuration
dave_graham Sure, but not as a persistently bound boot device
grealish In the BIOS I have boot support
dave_graham So, in my PowerEdge R610s, I can use the TOE feature (IIRC) of my Broadcom 5790s, but, really, not much to offer. Intel NICs/PHYS are much better IMHO.
grealish But that’s about it
dave_graham But, do you have boot support from VMware? That criteria is not supported, IIRC
grealish Okay, got all Broadcom :(
dave_graham Anyone can PXE boot a NIC (yes, that's supported) but using iSCSI to boot from a SAN is different
Guest_16 The boot support in the broadcom onboard Ethernets are...lacking? Requiring point of failure DHCP servers? Ick. Just my opinion. Have fun guys and girls
grealish Boot from iSCSI, that I can do with this adapter
dave_graham So, chances are, it won't work. Nothing to say you couldn't try, but I don't believe it will work
grealish Oh well, thanks. Anyways, GTG, see you next week ;)
dave_graham Cheers!