Dell MD3200i correct setup with IP Scheme and network map for faster Network speeds

Storage

Storage
Information and ideas on Dell storage solutions, including DAS, NAS, SAN and backup.

Dell MD3200i correct setup with IP Scheme and network map for faster Network speeds

This question is answered

Hello Everyone,

I have been working on this project for a while now. I have done lots of research and have contacted Dell multiple times. I need some help.

I have 2 sets of identical servers to be used for Hyper-V clusters. We purchased a Dell MD3200i in November, we also purchased a Dell 5424 switch and a Cisco SLM2048 switch. These servers have Real Tek network cards 2 each for iSCSI, Jumbo frames is on to 9K

The Dell MD3200i is dual controller SAN. The Dell 5424 is the iSCSI optimized switch and the Cisco is a 48 port switch. iSCSI is on its own network nothing else is connected to it. (There is a Iomega NAS and a Promise Tech Vtrak SAN with the domain controller all running with iSCSI)

Dell MD3200i  Controller 0 Port 0  192.168.130.101                    Controller 1 Port 0  192.168.131.101

                            Controller 0 Port 1 192.168.130.102                    Controller 1 Port 1   192.168.131.102

                            Controller 0 Port 2 192.168.130.103                    Controller 1 Port 2    192.168.131.103

                             Controller 0 Port 3 192.168.130.103                   Controller 1 Port 3    192.168.131.104

Cisco Switch has 1 VLAN #1 on it and only the 130 subnet items in it. There are 8 connections to the switch. 4 from Controller 0 and 1 from each of the servers. This has an IP address of 192.168.132.4 but that switch IP should not make any difference. Jumbo frames turned on

Dell Switch has 2 VLAN's #1 for the misc stuff like the DC, Vtrack and the NAS. Subnet 2 is 131 and has a total of 8 connectors in it. 4 from Controller 1, and 1 each from the servers. Jumbo frames turned on. IP Address of the switch is 192.168.132.2

Server Groups. Each Server node is running Windows 2008 R2 SP1 and running hyper-v clusters with Virtual Servers / file storage on them. 2 Realtek NIC's with Jumbo frames turned on to 9K ProdNode1 and ProdNode2 have 12 Core i7 CPU with 12 GB Memory, EXNode1 and ExNode2 have 12 Core i7 CPU with 24 GB memory.

                  ProdNode1 iSCSI-1  192.168.130.110                       ProdNode1 iSCSI-2  192.168.131.110

                 ProdNode2 iSCSI-1   192.168.130.120                       ProdNode2 iSCSI-2  192.168.131.120

                ExNode1 iSCSI-1         192.168.130.130                      ExNode1 iSCSI-2        192.168.131.130

               ExNode2 iSCSI-1         192.168.130.140                       ExNode2 iSCSI-2      192.168.131.140

iSCSI Discovery portal on ProdNode1 went to 192.168.130.101 from 192.168.130.110 and 192.168.131.101 from 192.168.131.110 Each of the other servers discovery portal were pairs like that.

iSCSI Targets Tab click that, add 8 sessions MPIO to each of the ports on the Dell MD3200i. For example I added 1 session from 192.168.130.101 using this. Add Session, --> Enable Mutlipath --> Advanced --> Local Adapter iSCSI --> Initiator IP 192.168.130.110 and Target IP 192.168.131.101. Repeated this on all servers. 4 Connections for each NIC.

The network is very slow. It was running around 2 to 4 MB / MIN and it should be much faster. At first we only had the Cisco Switch with only 1 VLAN and everything connected directly to it. I read somewhere that MPIO was the problem so I went through the same setup but did not check MPIO checkbox above. Once it was done, the network speed increased to 1.5 GB / Min but the Dell Modular storage wasn't happy and the Virtual servers would fail and not very stable connection.

What is wrong with my setup? Can I do direct attached and attach the Servers directly to the PowerVault and will that make it stable and allow for failover and NIC redundancy? I need to get this stable so I can move on with something else.

 

Thanks Ron

Verified Answer
  • You will want to change your IP scheme to this:

    rbramble
    EX-Node1             iSCSI-1                  192.168.130.130
    EX-Node1             iSCSI-2                  192.168.131.130
    EX-Node2             iSCSI-1                  192.168.132.140
    EX-Node2             iSCSI-2                  192.168.133.140
    Prod-Node1          iSCSI-1                  192.168.130.10
    Prod-Node1          iSCSI-2                  192.168.131.10
    Prod-Node2          iSCSI-1                  192.168.132.20
    Prod-Node2          iSCSI-2                  192.168.133.20
    Dell MD3200i      
    Controller 0 Port 0 - 192.168.130.101 Jumbo Frames 9K
    Controller 0 Port 1 - 192.168.131.101 Jumbo Frames 9K
    Controller 0 Port 2 - 192.168.132.101 Jumbo Frames 9K
    Controller 0 Port 3 - 192.168.133.101 Jumbo Frames 9K
    Controller 1 Port 0 - 192.168.130.102 Jumbo Frames 9K
    Controller 1 Port 1 - 192.168.131.102 Jumbo Frames 9K
    Controller 1 Port 2 - 192.168.132.102 Jumbo Frames 9K
    Controller 1 Port 3 - 192.168.133.102 Jumbo Frames 9K

    Your iSCSI sessions will then be:

    For EX-Node1:

    - 192.168.130.130 to 192.168.130.101

    - 192.168.130.130 to 192.168.130.102

    - 192.168.131.130 to 192.168.131.101

    - 192.168.131.130 to 192.168.131.102

    For EX-Node2:

    - 192.168.132.140 to 192.168.132.101

    - 192.168.132.140 to 192.168.132.102

    - 192.168.133.140 to 192.168.133.101

    - 192.168.133.140 to 192.168.133.102

    For Prod-Node1:

    - 192.168.130.10 to 192.168.130.101

    - 192.168.130.10 to 192.168.130.102

    - 192.168.131.10 to 192.168.131.101

    - 192.168.131.10 to 192.168.131.102

    For Prod-Node2:

    - 192.168.132.20 to 192.168.132.101

    - 192.168.132.20 to 192.168.132.102

    - 192.168.133.20 to 192.168.133.101

    - 192.168.133.20 to 192.168.133.102

    Member since 2003

All Replies
  • First, change your IP scheme back to the factory default; it was set like that for a good reason; it's the best practice:

    controller 0 port 0: 192.168.130.101

    controller 0 port 1: 192.168.131.101

    controller 0 port 2: 192.168.132.101

    controller 0 port 3: 192.168.133.101

    controller 1 port 0: 192.168.130.102

    controller 1 port 1: 192.168.131.102

    controller 1 port 2: 192.168.132.102

    controller 1 port 3: 192.168.133.102

    Then you upgrade to Intel or Broadcom NICs. Realtek NICs are acceptable for desktop computers, but if you're building an iSCSI network and you spend the money on decent switches for this iSCSI implementation, don't try to save a few dollars by using Realtek NICs (why do you think none of the server manufacturers (IBM, HP, Supermicro, Tyan, Dell, etc) use Realtek NICs on their server motherboards?).

     

    Then you plug the connections for 192.168.130.x and 192.168.132.x into one switch/vlan and 192.168.131.x and 192.168.133.x into the 2nd switch/vlan. This is the start of a proper iSCSI setup.

    Member since 2003

  • OK,

    If I set everything back to default? And replace the NIC's with other cards? How do I set up the servers? What connections do I set them up to? Any detailed doc's on this?

     

    Thanks

    Ron

  • You want 4 nics in each host for iSCSI to these arrays.   Each runs on a different subnet.   Generally you need a dedicated switch for each subnet or at least a dedicated Vlan on a set of switches.

    Powervault arrays do not do well with multiple iSCSI ports on the same controller on the same subnet. 

     

  • Hello,

    I have looked at more documentation and need to get an idea of this.

    I am looking at the http://support.dell.com/support/edocs/systems/md3200i/en/DG/HTML/index.htm

    Appendix C, Figure 1 http://support.dell.com/support/edocs/systems/md3200i/en/DG/HTML/appendic.htm#wp1186780

    is this information right? Under the TCP Connections T01 shows IP_Addr_If1 / IP_Addr_C0_P0  T02, TO3, T04 shows connecting to Controller 1. However T11: through T14 shows connecting to Controller 1

    Thanks

     

  • The diagram is right for a two nic per host setup.    Really,  don't cheat your self on these array,  use 4 nics per host.   You get 4 x 1gb bandwidth.

    I say that,  but I must admit,  I plan to only have two nics in each host for quite sum time even when we go to our new storage servers which could easily be configured for 4.

     

  • Hello,

    I have updated my settings. It is much faster but is it as fast as the equipment can go? I have seen faster when the MPIO was not enabled but the SAN wasn't stable.

    I have 2 Clusters with 2 Nodes on each Server. Each server is running Windows 2008 R2 SP1. There are 2 NIC's in each Server dedicated to iSCSI. I have 1 Switch that is dedicated to iSCSI. There are 3 VLANS on the switch, Jumbo frames Enabled, Not Tagged. All NIC's in the Servers are running Jumbo Frames 9K.All 130.xxx address are on VLAN 130, All 131.xxx are on VLAN 131. The switch and my other SAN and a NAS and the 2 Domain Controllers are also running on the iSCSI switch and their address is 192.168.135.xxx and they are all on VLAN 1.

    I had a Dell 5424 Switch but lost access to the Web Management portal via the internet browser so I went back to the other single switch. I have another Dell 5424 switch that I bought coming in. Can anyone recommend how to setup the switches including access from the Web portal not the console cable?

    Normal 0 false false false EN-US X-NONE X-NONE MicrosoftInternetExplorer4

    EX-Node1             iSCSI-1                  192.168.130.130

    EX-Node1             iSCSI-2                  192.168.131.130

    EX-Node2             iSCSI-1                  192.168.130.140

    EX-Node2             iSCSI-2                  192.168.131.140

    Prod-Node1          iSCSI-1                  192.168.130.10

    Prod-Node1          iSCSI-2                  192.168.131.10

    Prod-Node2          iSCSI-1                  192.168.130.20

    Prod-Node2          iSCSI-2                  192.168.131.20

    Dell MD3200i       Controller 0 Port 0                  192.168.130.101 Jumbo Frames 9K

                                    Controller 0 Port 1               192.168.131.101 Jumbo Frames 9K

                                    Controller 0 Port 2               192.168.130.103 Jumbo Frames 9K

                                    Controller 0 Port 3               192.168.131.103 Jumbo Frames 9K

                                    Controller 1 Port 0               192.168.130.102 Jumbo Frames 9K

                                    Controller 1 Port 1               192.168.131.102 Jumbo Frames 9K

                                    Controller 1 Port 2               192.168.130.104 Jumbo Frames 9K

                                    Controller 1 Port 3               192.168.131.104 Jumbo Frames 9K

    ISCSI Settings Example. Each Server had 2 Discovery Portals, Add a MPIO Session and then 4 MCS Sessions per each TCP Target Session.

    Discovery              192.168.130.101 -- 192.168.130.130

                                    192.168.131.103 -- 192.168.131.130            

                                    Source                                    Target

    Target (1)              192.168.130.130                                 192.168.130.101

    MCS Sessions       192.168.130.130                 192.168.130.103

                                    192.168.131.130                 192.168.131.101

                                    192.168.131.130                 192.168.131.103

    Target (2)              192.168.131.130                 192.168.131.102

    MCS Sessions       192.168.131.130                 192.168.131.104

                                    192.168.130.130                 192.168.130.102

                                    192.168.130.130                 192.168.130.104

    This is pretty long but it includes the settings.

    Can anyone recommend an action to make it faster or reconfigure the Dell 5424 switches so they will be optimized iSCSI and connect via the Web Portal? Thanks for your assistance.

  • You will want to change your IP scheme to this:

    rbramble
    EX-Node1             iSCSI-1                  192.168.130.130
    EX-Node1             iSCSI-2                  192.168.131.130
    EX-Node2             iSCSI-1                  192.168.132.140
    EX-Node2             iSCSI-2                  192.168.133.140
    Prod-Node1          iSCSI-1                  192.168.130.10
    Prod-Node1          iSCSI-2                  192.168.131.10
    Prod-Node2          iSCSI-1                  192.168.132.20
    Prod-Node2          iSCSI-2                  192.168.133.20
    Dell MD3200i      
    Controller 0 Port 0 - 192.168.130.101 Jumbo Frames 9K
    Controller 0 Port 1 - 192.168.131.101 Jumbo Frames 9K
    Controller 0 Port 2 - 192.168.132.101 Jumbo Frames 9K
    Controller 0 Port 3 - 192.168.133.101 Jumbo Frames 9K
    Controller 1 Port 0 - 192.168.130.102 Jumbo Frames 9K
    Controller 1 Port 1 - 192.168.131.102 Jumbo Frames 9K
    Controller 1 Port 2 - 192.168.132.102 Jumbo Frames 9K
    Controller 1 Port 3 - 192.168.133.102 Jumbo Frames 9K

    Your iSCSI sessions will then be:

    For EX-Node1:

    - 192.168.130.130 to 192.168.130.101

    - 192.168.130.130 to 192.168.130.102

    - 192.168.131.130 to 192.168.131.101

    - 192.168.131.130 to 192.168.131.102

    For EX-Node2:

    - 192.168.132.140 to 192.168.132.101

    - 192.168.132.140 to 192.168.132.102

    - 192.168.133.140 to 192.168.133.101

    - 192.168.133.140 to 192.168.133.102

    For Prod-Node1:

    - 192.168.130.10 to 192.168.130.101

    - 192.168.130.10 to 192.168.130.102

    - 192.168.131.10 to 192.168.131.101

    - 192.168.131.10 to 192.168.131.102

    For Prod-Node2:

    - 192.168.132.20 to 192.168.132.101

    - 192.168.132.20 to 192.168.132.102

    - 192.168.133.20 to 192.168.133.101

    - 192.168.133.20 to 192.168.133.102

    Member since 2003

  • Hello,

    I think I am done messing with this.

    I used the above configuration but I also added both Dell 5424 switches. Switch #1 is running the VLAN for the 130/132 subnet and the 2nd switch is running the VLAN for the 131/133 subnet. I set up both switches identically with 3 VLAN's and only plugged in the cables  into the separate ports associated with the VLAN for that switch. I have a NAS and a Vtrack SAN on the same network but put them on their own subnet.

    My test was a 21 GB backup to an LTO4 tape drive. When I first started it was almost 2 hours. Last night it was 11 minutes.

     

    Thanks for the assistance

    Ron

  • What are the default gateway and subnet mask settings for this IP settup? I'm a complete network newbie. Thank you.

  • what if you had a r2 cluser with 8 nic ports

    would it be

    node1

    iscsi-1       192.168.130.x

    iscsi-2        192.168.131.x

    iscsi-3         192.168.132.x

    iscsi-4         192.168.133.x

    node2

    iscsi-1       192.168.130.x

    iscsi-2        192.168.131.x

    iscsi-3         192.168.132.x

    iscsi-4         192.168.133.x

  • Tried switching to MRU to see if this is an MPIO issue but then the writes max at 110MB/s as expected but the read is still stuck ~2MB/s

    Thanks Jonathan

  • <deleted post and started a new thread>