Our featured guest will be Todd Muirhead, VMware Performance Engineer. He will be discussing his work with the Dell TechCenter around SAP and Microsoft Exchange 2010 on VMware vSphere 4.

Technical Community - Background Reading



Chat Transcript

Dell-KongY Was there anything in particular that you were interested in- the SAP or Exchange 2010 piece?
robinberliner Not really, I am coming to learn
Dell-KongY Excellent- I'm hoping to post a wiki page on my experience w/ 2010 and hopefully evolve it based on the community's participation
Dell-KongY Todd will discuss aspects around SAP
Techanalyst Yes, I am interested in migration/upgrades paths best practices, for instance...
robinberliner Yes, I am also interested in best practices
Techanalyst Does Dell have tools to manage these?
Dell-KongY @techanalyst and @robinberliner- noted. Great questions.
Dell-KongY We are looking at best practices- do I use VMware or Exchange? There are other considerations such as Microsoft recommending not to virtualize the Mailbox server for Exchange 2010 as well.
Techanalyst I am also interested in Exchange 2010 from an Ad perspective. Any light that can be shed as it relates to upgrades/migrations.
Dell-KongY Unfortunately, those topics will not be covered in too much detail in this chat.
Dell-KongY This chat revolves more around Sap 3-tier performance testing and VMotion testing on vSphere & Exchange 2010 performance testing on vSphere
Dell-KongY But your questions and engineering perspective is exactly why we have these chats
Techanalyst Okay, I am in. I am interested in applications performance
Dell-KongY The best practice around migration is a great topic that we are evaluating- there are many non-technical formalities that encompass it :)
Techanalyst I am aware, yes. I should like to understand it from a Dell perspective. Perhaps, another chat soon?
Techanalyst Is Robin a Dell person or It?
Dell-KongY I will add the Update/migration of Exchange as a possible topic
Dell-KongY I'm not sure if Robin is a Dell person or not.
Dell-KongY Most Dell folks on the chat will Id with Dell somewhere in there username
Techanalyst One note to take is that I am always interested in highly technical chats. Especially from a Dell hardware perspective. Lows, for example, how Sap 3-tier perf is facilitated by the HW Config.
Techanalyst So, I do not recall who the speaker/chat lead is going to be...
Techanalyst Is s/he from SAP, Microsoft, or...
Dell-KongY Todd Muirhead, Performance Engineer from VMware is the featured guest
Techanalyst Cool, thanks.
Dell-KongY Todd and I have worked together on many projects when Todd was a Dell engineer and even now when he's a VMware engineer
Dell-KongY He will share his Sap on vSphere performance experience and I will preview initial Exchange 2010 performance on vSphere 4
Techanalyst Since we still have another 1/2 hr. if you like I can return and give you a break
Dell-KongY tech- i'm cool either way. if you have other questions around virtualization, performance., x86, we can discuss
Techanalyst Do you guys pre-sales, lab, or integration?
Techanalyst *Are you guys...
Techanalyst Applications, hardware or both?
Dell-KongY I'm an evangelist in the Dell TechCenter focused on virtualization, blades, and power and cooling
Dell-KongY I came to this position from Dell Pg- Office of the CTO x86 performance analysis and the Virtualization Solutions Engineering team
Dell-KongY We are the http://www.delltechcenter.com/ - generating content, engaging in the discussion forum
Dell-KongY We try to leave sales and marketing at the door
Dell-KongY Let's wait a few minutes to get started
Dell-KongY Today's chat is centered around Sap and Exchange 2010 performance on Vmware vSphere 4
Dell-KongY Our featured guest is Mr. Todd Muirhead, VMware Performance Engineer, and @virtualtodd for Twitter folks
virtualTodd I'm actually work out of the VMware Austin office, so I get to stop by the real live Dell TechCenter and work with Kong
Dell-JeffS Todd's actually been in the lab quite a bit lately... care to share what you've been working on?
virtualTodd Kong and I have done lots of work with Exchange on ESX in the past and have started looking at Exchange 2010 more recently
Dell-d_glynn Hey kong, i expected a bigger crowd for this topic
virtualTodd I have also been working with Kong on some Sap on vSphere testing in the Dell TechCenter lab as well to measure 3-tier performance over the past couple of months as well
sixth Just as a reference, I am running three exchange 2003 servers on vSphere with no issues, trying to push for 2010
virtualTodd In terms of Sap testing, it has been really great to get to use some of the latest Nehalem based servers and Equallogic storage
virtualTodd I have basically done two types of SAP testing in the DellTechCenter lab.
virtualTodd One was on Sap batch job performance - Link coming
Techanalyst Will you be giving the scenario/case study for your chat?
virtualTodd http://blogs.vmware.com/performance/2010/02/sap-batch-job-performance-on-vsphere.html
virtualTodd And the other is some 3-tier scaling testing using the SAP SD Benchmark
virtualTodd That one hasn't been published yet because I just got the final set of numbers yesterday!
virtualTodd Techanalyst - Which one do you want to talk about first?
JasonPowell I have a customer interested in doing SAP on EqualLogic but they want to know who's doing it already or have tested it
Techanalyst Whichever you prefer... They are both important
virtualTodd A bit of background - Sap on ESX is actually fairly popular in smaller configurations
virtualTodd We have lots of customers running in what is known as 2-tier configurations, where the database and application tiers are on the same virtual machine
virtualTodd In order for larger configurations to be able to run, it becomes necessary to split out the app and db and make it a 3-tier environment
virtualTodd We do have some customers doing this as well, but there really hasn't been much published in this area.
virtualTodd So the SAP batch job testing used a 3-tier environment with 1 and then 2 application servers’ virtual machines running against a database virtual machine.
virtualTodd SAP will basically take the number of application servers assigned to a batch job and then use as many threads as possible to complete the task.
virtualTodd This makes it a very parallel job with lots of network traffic between the application and database virtual machines.
virtualTodd So this batch job was very CPU and network intensive - but was still able to achieve almost the same performance as similar physical only setup
virtualTodd The one thing that was somewhat surprising was that I had to use more work threads on the virtual application servers than on the physical application servers to achieve the same performance
Techanalyst Are we talking about E-ps600? Ip San? E-iscsi? Can you please present the environment in which the lab was conducted? Is this okay to ask?
virtualTodd More details on lab - no problem
virtualTodd The storage is 4 EqualLogic Ps5000xv ISCSI arrays
Dell-KongY There are 64x 146GB drives in Raid 5 with 4 hot spares
virtualTodd The servers are PowerEdge M710 with x5570 Nehalem processors
sixth What kind of switches?
virtualTodd Ethernet :)
virtualTodd Kong - What are the model numbers on the switches?
Dell-KongY PowerConnect 6324s
sixth Brand/model?
sixth Haha
sixth Stacked I am assuming?
Dell-JeffS Yes
Dell-KongY On the blades- IO Modules are M6220s
Dell-JeffS 6248's I thought
Dell-KongY I defer to Jeff our San guru :) 6348s are the model # on the PowerConnects :)
Techanalyst Expand of your Raid 5 choice, please
virtualTodd The disk I/O for the SAP tests isn't as intensive as the testing that we did with Exchange 2007 last year as it turns out
JasonPowell Raid5 vs. 50 ... interesting
Techanalyst Hardware wise this was a completely homogeneous environment? Meaning all Dell harware...
Techanalyst Correct?
virtualTodd The hardware was in the DellTechCenter lab, so yes it was all Dell hardware
dhelm Is SAP running on WIN/SQL, WIN/ORC, or UNX/ORC?
Techanalyst Jason why is r-50 interesting to you?
virtualTodd In the batch job testing Sap was running on SLES Linux with Oracle DB
virtualTodd The 3-tier testing was with WIN/SQL
virtualTodd The 3-tier tests used the SD benchmark and utilized 3 app server VMs and 1 DB VM
Techanalyst Virtual - that's cool. I am just looking at the empirical hardware landscape!
JasonPowell techanalyst - because most everyone I know running EqualLogic does not run Raid5 but rather 50
virtualTodd I started off with 4 x 2vcpu virtual machines and was able to get about 1300 users
Techanalyst Oracle 11g, 10g??? Earlier?
virtualTodd This was only using 8 of the available 16 logical threads on the server
Techanalyst Thanks Jason
virtualTodd It was Oracle 10g (I don't remember the dot release level off the top of my head)
virtualTodd So in order to use the other 8 threads (the hyperthreads) I increased the VCPUs to 4 for all of the VMs
Techanalyst That's cool, version is good
virtualTodd This allowed me to increase the number of users to about 1700, an increase of roughly 30%
virtualTodd At this point the system was around 90 to 95% utilized
virtualTodd So to show that more users could be added by scaling out to another host....
virtualTodd I then used VMotion to move two of the app server virtual machines to another blade server.
virtualTodd I was then able to push the number of users to about 2600 users, or about 2x the original 1300 users I was getting.
virtualTodd Showing that scaling with 3-tier environment on vSphere works more or less as expected with additional performance as additional resources are added.
Techanalyst Were all the users on the same Ou?
dhelm Even though network communications would be higher in a 3-tier setup. Wouldn't you potentially get better performance by splitting the DB off from the application? Are you planning to test that scenario as well?
virtualTodd The SD benchmark allows for up to 1000 users per client, and I have three clients setup to support up to 3000 users in my test - all users were on the same landscape
Techanalyst Thanks
virtualTodd dhelm - there is some additional overhead that is caused by going to 3-tier due to additional network,
virtualTodd So if you were to compare a 2-tier vs. 3-tier with the same amount of resources, the 3-tier would be a little bit lower due to this
Techanalyst Were you able to bench the per user activities?
Dell-KongY There will be a video of the Sap VMotion that will be coming out soon
dhelm Yes, but that is our current phys setup. We will be doing p2vs.
virtualTodd Of course going to 3-tier allows you to have more resources and hence more performance in the end
Techanalyst What was your bandwidth?
virtualTodd If your SAP will fit in 2-tier, then the performance will most likely be better staying in 2-tier
virtualTodd Once you outgrow it, and you need 3-tier it makes sense to move to it
virtualTodd All connections were 1-gigabit Ethernet
dhelm Kong - Adaptive Computing Controller used as well with VMotion?
Dell-KongY dhelm- we just performed a manual VMotion
dhelm I have done this as well with our proof of concepts. I was just wondering if you guys had used the Acc and found any benefit to using it.
Techanalyst *dhelm you probably should publish here...
virtualTodd dhelm - I have found that VMotion works very well with Sap - Did you find the same thing ?
Techanalyst What tools did you both use to test performance?
virtualTodd There is currently some work underway to test larger 3-tier environments
dhelm VMotion and storage VMotion both worked very well. We have NetApp storage on the back end and you just have to make sure that your iGroups are setup so each ESX host can see the storage.
virtualTodd The SAP SD benchmark - or the Sap Sales and Distribution benchmark - http://www.sap.com/solutions/benchmark/sd.epx
Dell-KongY tools for testing performance – IO meter -> general storage, Load Generator 2010 -> Exchange 2010
virtualTodd The SD benchmark is not for the faint of heart to setup and use - but it is the standard in the Sap world
Dell-KongY Jetstress for disk subsystem for Exchange 2010
Techanalyst Did you guys deploy any load balancers?
virtualTodd To measure performance I used esxtop, SD benchmark results, and monitored the SAP queues and work threads
virtualTodd No load balancers, the SD benchmark was setup to evenly distribute load across the App servers
virtualTodd Kong - We are nearing the end and we didn't even get to Exchange 2010!
Dell-KongY Wow- an hour passed by and no Exchange 2010
virtualTodd Are we going to have another chat to focus on that?
Dell-KongY Another day then
Dell-KongY For sure
dhelm I was just going to ask if you had any papers on Exchange
Dell-KongY I want to thank Todd and all the chat participants
Dell-KongY It was a lively discussion
Dell-KongY dhelm - not on Exchange 2010
dhelm 2007
virtualTodd Any other SAP on vSphere questions? I can hang around for a few more minutes
Techanalyst How many app servers did you say? how large was the farm in GBs?
virtualTodd 3 app server virtual machines, 1 DB VM
Dell-KongY http://www.delltechcenter.com/page/microsoft+exchange has links to whitepapers around 2007 on Esx
dhelm thx
virtualTodd I believe that the DB was about 200 GB -
Dell-KongY http://www.dell.com/downloads/global/solutions/virtexchange2k7_finalfrontier_vmworld2007.pdf
durkinr I only came for Exchange 2010...so go ahead and reschedule!
virtualTodd I will have more details in blog post that comes out in the next week or so
Dell-KongY that was presented at VMworld 2007 by Todd and me
Dell-KongY I will be posting a wiki page on Exchange 2010 on VMware vSphere shortly