By Xin Chen, Garima Kochhar and Mario Gallegos. August 2012.
Almost all clusters, no matter the size, need an NFS based solution. In some clusters this storage is used only for applications and home directories. In others depending on the size of the cluster and the IO requirements, it can be used for processing temporary files as well. Every system admin knows NFS, but in our experience, tuning NFS is non-trivial. You already have the servers, the storage and the software. How do you get the best performance and reliability out of your configuration? And how do you get up to 4000MB/s throughput from NFS? Keep reading!
NSS-HA is a line of optimized NFS based storage solutions for HPC configurations that also provide High Availability. Including the latest solution described here, three versions of NSS-HA solutions have been released since 2011. With the introduction of the latest version of NSS-HA (called NSS4-HA and described in this article), the NFS servers have been upgraded to take advantage of several new technologies that promise to improve IO performance.
We’re going to throw some model numbers at you now. This is to easily and simply explain what new technologies we’re talking about. If you’re familiar with this, skip ahead to the next paragraph. These technologies have been released with the Dell PowerEdge R620 server. It features the Intel Xeon E5-2600 series processors (based on the Intel micro-architecture codenamed Sandy Bridge-EP) and provides enhanced systems management features and lower power consumption when compared to the previous 11th generation Dell PowerEdge servers. The integrated PCIe Gen-3 I/O capabilities of the latest Intel Xeon processors allow for a faster interconnect using the 56 Gb/sec fourteen data rate (FDR) InfiniBand adapters. For 10 Gb Ethernet solutions, an onboard Network Daughter Card that does not consume a PCIe slot is now an option. The 1U PowerEdge R620 has enough PCIe slots to satisfy the requirements of the NSS-HA solution and allows for a denser solution. Additionally, the PowerEdge R620 provides increased memory capacity and bandwidth. All these factors combine to provide better IO performance as described below. The storage subsystem of this release of the NSS-HA remains unchanged.
For readers familiar with the NSS-HA solutions, Table 1 gives an easy to read way to see what’s new. It also allows you to decide if you need an upgrade at all. Note that there are significant changes in configuration steps between the NSS2-HA and NSS3-HA releases, while there are few configuration changes between the NSS3-HA and NSS4-HA releases. With the help of the PowerEdge R620 and FDR InfiniBand network connection, the NSS4-HA solution now achieves sequential read peak performance up to 4058 MB/sec! The Performance section at the bottom of this article gives more detail on the IO performance of this configuration compared to the previous generation.
Table 1. The comparisons among NSS-HA Solutions
Compared to the NSS3-HA solution releases, the biggest change in the NSS4-HA release is that the new Dell 12th generation PowerEdge R620 server is deployed as an NFS server, while the Dell 11th generation PowerEdge R710 server was used as the NFS server in the two previous releases.
The Dell NSS-HA solution is designed to provide storage service to HPC clusters. Besides providing high availability and reliability, it is also essential for a storage solution to deliver excellent I/O performance for HPC clusters. The PowerEdge R620 leverages current state-of-art technologies to enhance existing network and disk I/O processing capabilities, compared to PowerEdge R710. The key features of the PowerEdge R620 are listed below. These features position the PowerEdge R620 to be a better performing platform and better performing NFS server in NSS-HA than the PowerEdge R710:
Due to the many powerful features of the PowerEdge R620, the current NSS-HA release provides significant I/O performance improvement:
The following figures show the comparisons between NSS3-HA and NSS4-HA. Note: NSS3-HA and NSS4-HA have the exact same storage subsystem.
Figure 1. IPoIB large sequential write performance: NSS4-HA vs. NSS3-HA
Figure 2. IPoIB large sequential read performance: NSS4-HA vs. NSS3-HA
Figure 3. IPoIB random write performance: NSS4-HA vs. NSS3-HA
Figure 4. IPoIB random read performance: NSS4-HA vs. NSS3-HA
Figure 5. IPoIB file create performance: NSS4-HA vs. NSS3-HA
Figure 6. IPoIB file stat performance: NSS4-HA vs. NSS3-HA
Figure 7. IPoIB file remove performance: NSS4-HA vs. NSS3-HA
For detailed information about Dell NSS4-HA solution, please refer to “Dell HPC NFS Storage Solution High Availability (NSS-HA) Configurations with Dell PowerEdge 12th Generation Servers.” For the detailed Dell NSS4-HA configuration guide, please refer to the attachment of the blog.
The HPC Engineering team at Dell has been focused on NFS with High Availability for some time, particularly