Greetings folks

I have an MD3000i acting as storage for a PE2950; below is my observation:
PE2950 file copy local drive to MD - ok throughput, up to 70% of GbE, about roughly the same when file copy is from MD to local drive
Client on the public LAN file copy from share on PE2950 (local drive), ok throughput also

However, when the client maps to the file share, and the share happens to be from the MD - I obviously see roughly equal utilisation on both the public LAN and iSCSI LAN NICS (both using the server onboard NICs); but performance is very bad like 7% of GbE.

First thing that came to mind was that the PE2950 couldn't drive both NICS at the same time, seeing that testing from client > server local disk or server local disk > MD worked ok; but the PE2950 is not ancient hardware either (4GB RAM, 2.x GHz), so it can't be that bad?

Thought it might be due to interrupt sharing, but it's been ages since I had to tweak that on a server, but anyway I've tried tweaking those but it does not seem to help.

Does anyone have any leads on where to go to? Firmware is latest on the MD3000i. (15 x 600GB 15K SAS), also updated the SAS disk firmware.

Thanks!

Hmm... that kind of performance drop does sound quite weird. Are you using plain windows file sharing? Are the two NICs on separate subnets? Did you perchance take a look at the system utilization to see if anything was getting overloaded?
Any chance of looking at a wireshark trace?

Yep, it's plain windows file sharing - actually I noticed it first when networker was doing its backups over lan, the backup throughput for the client was very bad. To narrow down the problem, I tested with plain file sharing and encountered the symptoms above.

NICS are on different subnets (public, and the iSCSI network as per the quick setup poster; no other hosts connected on the iSCSI side (procurve 2910al-24G switch).

I haven't had a chance to look at the trace, but anything in particular to look out for? SYN/ACK retransmits or something? Haven't used it much so I'm quite green there :D

CPU and mem usage was ok, based on task mgr performance monitor.

Server 2008 R2? I am thinking some odd O/S bug maybe?
Are you using O/S MPIO or are you using Dell's MPIO. When ever I am on with a tech and I have the O/S MPIO loaded dell rips it out and installs their own.

"Server 2008 R2? I am thinking some odd O/S bug maybe?
Are you using O/S MPIO or are you using Dell's MPIO. When ever I am on with a tech and I have the O/S MPIO loaded dell rips it out and installs their own."

Indeed - I would definately use Dell's own MPIO as you point out. I have (2) pe1950 running w2k3 on MD3000i for 2 1/2 years with no speed issues...

W2k8 is a different deal, but I would have to believe it is either a network port setting (full/half duplex or 100/1000, etc...). If you have ruled that out then I would look at the iSCSI MPIO, etc.

Also, are you trying to use Jumbo packets? If those are not set correctly you can get wierd performance issues.. If thats the case I would just disable the jumbo packets across the board and see how it runs to rule that out.

I'm quite green with wireshark, but a before and after trace might give some idea. However, I doubt that that is the case as the NIC is not getting overwhelmed. It does seem to be some weird OS bug. Did you check the event logs for any errors or timeouts?

mrokkam, JOHN and Sean - thanks for the insights and suggestions.

OS is Win2K3 R2 x86 and come to think of it, I did have the MS iSCSI initiator before getting the MD3000i; but can't recall now if the MPIO was using MS or Dell's; will do a clean install.
On jumbo frames, I set it on the MD3000i as 9000, on the NIC I recalled playing with different values (4000+ included) but didn't get much difference in results. Will try with all parties set to default 1500.

Event logs are clean, nothing in particular about disk or LAN errors.

- to check MPIO
- turn off jumbo frames totally along the entire path, makes some sense since the client accessing the file server may not even use jumbo frames
- wireshark trace
- blow 2003 away and replace with 2008 :)

Thanks again all for the help, I've got next week's work all planned out now. Have a good weekend :)

Jumbo frames is interesting. We tested it with esx 3.5 and 4.0, and it actually decreases our IOPS quite a bit, nothing like your seeing though. Because of my experience on VMware I never even tried it on our Windows Server attached sans.

We have two MD3000i's with attached Vmware hosts and two with with Windows Server attached hosts those being a mix of 2003 and 2008 server. Now I will caution you some on running throught put testing as sustained throughput isn't always impressive, but the dang IOPS we are getting is darn impressive considering the low cost of these iSCSI sans. IOPS will be the overwhelming performance factor in business server application use. IE: Exchange, SQL, file and print, ect..ect...

One note on the Jumbo Frames? We use pretty cheap GIGE Switches on our iSCSI san networks, but they do support large frames up to the 9K-ish limits. So other "better" switches may perform better on it.

With most all business sevrer applications on sans? You would never want to trade the a better sustained throughput number for lower overall IOPS.

Hi everyone,

I experience same poor performance. I have 2 MD3000i with 8 SATA 7,2k 1To disks and RAID 6 configured. Both are connected on a Windows 2003 x64 server (R710). I use Broadcom internal NIC and BACS software. I've enable Jumbo Frames (MTU 9000) on both Broadcom iSCSI and MD3000i. I use the Microsoft Initiator but configured with Broadcom iSCSI Adapter (need a configuration).
I really don"t understand what's going on. When i just copy a file from attached disk to MD3000i it seems fast but when i do defrag on the drive on the MD3000 it's a nightmare.
If some one have a solution or an advice.

Thanks.

I have never tried a defrag on a disk on the san. Is it just taking for ever? I mean they took forever when I was on direct attached too. So long, I generally could never run them.

Defrag permits to increase perf for my backup solution (stored on the MD3000i drive). Anyway, cylces consist in read/write operation and seem really slow.
The fact is that I do experience the same poor performance than a lot of people here. I have configured iSCSI link with dedicated tools (Broadcom tool) and i really don't see any differences.
Maybe someone knows how to increase perf by tuning OS?

Most all users eventually figure out where the issue is. But, troubleshooting this is going to require much more information.

Even though I don't think this is the issue.
8 Drives, slow cruddy sata drives, and RAID 6 may not yield results good enough for many situations.

We have a couple of windows x86 servers connected by a single NIC to single controller MD3000i's with 15 slow cruddy sata drives in the disk group. This should be as slow as slow can be. I would be willing to try some operations on it of your choosing to compare numbers?

ecolas - what kind of perf. do you get? If you're using SATA drives exclusively, there's an issue with MD3000i+SATA drives - more at http://www.delltechcenter.com/thread/3686431/MD3000i+and+write+perf

I guess there is a reason they really push you away from SATA on these. We use the SATA MD3000i's to back up on and DR only. The snap shots and disk copies work well for us. We can disk copy a couple a TB's to SATA in a few hours though, without to much degredation on the array. In initial testing, we found in general the array to not be fast enough unless we put 15 drives in the disk groups though.