Hi. I have a multi-tiered Equallogic SAN where the server (blade) access switches are Dell M6220s and the tray side access switches are Dell PC6248s. Access switches aggregate to a common distribution - Nexus 7018. Jumbo frames are enabled on access and distribution tiers. Flowcontrol is only enabled on the access switches. Is it also a requirement to have flowcontrol enabled on the distribution tier?
From the host to the array, flowcontrol should be end-to-end. So if the distribution tier is passing iSCSI traffic between the host and array, yes, it should be enabled.
Follow me on Twitter: @joesatdell
Thank you Joe. So on the distribution flowcontrol should be enabled as "receive on" or "transmit on" as well?
"Receive on" should be all that is required.
Although in the answers it is advised to enable flow control, and if doing so it has to be enabled end-to-end: I do see that you have PCT62xx switches in the setup. For flow control to work good the switches do need buffering capacity (so they can buffer some data when one end can't handle the rate of incoming data and thus sending a pause frame to hold off data) and the PCT62xx series is lacking (much/usable) buffering capacity.
As member of the Dell networking support team in EMEA we have found that it can be better NOT to use flow-control when PCT62xx is involved in an iSCSI environment: but testing what is best in your specific setup is the best way to find out.
When dealing with tiered switches it gets complicated fast. There are some specific configuration considerations for Nexus switches as well. We find that flowcontrol (FC) to the servers and enabled on the array generally works best. Especially on large reads (backups) it's easy to overwhelm the servers input buffers. The server will pause the port. As mentioned the PCT62xx don't have large port buffers so they will fill quickly, especially with Jumbo Frames. For scaling, generally speaking we don't like core switches being in the middle. Having the arrays as close as possible to the servers generally works best. Not that you can't use a large core switch, but careful configuration is needed to make sure it does not add in unwanted latency. I.e. core switch gets busy and doesn't pass iSCSI traffic in a timely manner. iSCSI is very latency sensitive. Core switches that also do VOIP tend to set preference to that vs. iSCSI. I've dealt with a number of customers who started out in that model, but as they grew it did not scale well. For smaller configurations it can work just fine.
The EQL array will never send a pause frame. But if FC is enabled it will honor one from the switch. When the array reaches saturation, it will use congestion controls, i.e. reduces TCP window sizes on specific iSCSI sessions vs. shutting down the whole port.
If you haven't done so already I would suggest opening a case with Dell to review the switch configuration of both the Dell PCT62xx and the Nexus. To insure they are configured to best practices.
Also I would suggest not using FEX's on array or iSCSI server ports. They offer no buffering, nor local forwarding, all packets must traverse up to core then back down to FEX.