For customers looking to deploy an advanced NFS Storage Solution (NSS) configuration that has higher aggregate capacity and better aggregate performance, the Extra Large or XL configuration with two mount points is now an available option. This configuration increases the total usable capacity up to 160TB (across two file systems) while continuing to provide the high availability and performance of the existing configurations. The XL configuration builds upon the existing Dell NFS Storage Solution with High Availability (NSS-HA) Large configuration. Details on the NSS-HA solution are available at http://content.dell.com/us/en/highered/d/business~solutions~whitepapers~en/Documents~dell-hpc-nssha-sg.pdf.aspx This blog post describes the XL configuration, performance, and HA functionality. For an introductory blog on NSS-HA, refer to the overview blog post.

Figure 1 shows the XL configuration.

The XL configuration has the following components:

  • Two PowerVault MD3200 arrays and six PowerVault MD1200 arrays: 1 MD3200 and 3 MD1200 per file system.
  • Two dual-port 6 Gbps SAS cards in each of the two PowerEdge R710 servers.
  • Two 6 Gbps SAS connections from each PowerEdge R710 server to each MD3200 storage array.
    • Each R710 is connected to both RAID controllers on each MD3200 storage array.
    • A total of eight 6Gbps SAS connections from the two R710s to the two MD3200 storage arrays.
  • Each file system comprises of one MD3200 expanded using three MD1200 arrays.
    • Each MD3200 and MD1200 array contains twelve 2TB 7200rpm NL SAS disks configured in a RAID6 (10+2) LUN.
    • A total of ninety-six 2TB NL SAS disks (forty-eight 2TB NL SAS disks per file system).
  • The four LUNs associated with each MD3200 storage array are combined into a Linux logical volume using LVM and formatted as an XFS file system as shown in Figure 1.
    • There are two XFS file systems in total. They are independent and have a separate namespace.
  • The software components are same to the NSS-HA Large configuration. The High Availability component is provided by the Red Hat Enterprise Linux 5.5 Cluster Suite software.



High Availability in the NSS-HA XL configuration

Compared with NSS-HA Medium and Large configurations, we have one more MD3200 storage array to manage for the XL configuration. To make our HA solution simple and consistent, our strategy was to introduce two active/passive pairs for the XL configuration to implement the HA feature:

  • Pair 1: server 1 hosts file system 1, server 2 is standby for that filesystem.
  • Pair 2: server 2 hosts file system 2, server 1 is standby for that filesystem.

When server 1 suffers a catastrophic failure, file system 1 originally hosted by it will fail over to server 2. At that time, server 2 hosts both file systems. In other words, there are two active NFS mount points exported by server 2. This strategy has been verified in our lab. The pros and cons are:

Pros:

  • Easy to understand and analyze:
Two * Large configurations = XL configuration.
  • The system availability is enhanced.
  • Simple method to double the usable storage capacity from the Large configuration to the XL configuration.

Cons:

  • More complicated NFS server cluster configuration and management;
  • When an NFS server fails, the performance of the healthy server may be affected as the server will not only handle its workload but also the workload transferred from the failed server. This impact is analyzed and quantified in the performance evaluation section below.



Performance Evaluation

The major difference between NSS-HA XL configuration and Large configuration is that one more MD3200 storage array expanded using 3 MD1200 arrays and two more 6 Gbps SAS cards are deployed for the XL configuration. We conducted two sets of experiments to explore the performance of NSS-HA XL configurations:

  • Two-server: each server has a NFS mount point, and the two servers share the same public network to the clients. This is a normal case in which no failure occurs.

  • Single-server: a single server has two NFS mount points. This case arises when one server suffers a failure, and its NFS service fails over to the other server. The remaining server hosts both file systems at this point.

For each set of experiments, we evaluated the performance of the XL configuration with two different network technologies to the NFS clients: InfiniBand (IB) and 10 Gigabit Ethernet (10 GbE). We analyzed the sequential I/O performance with a benchmark tool (Iozone). We wanted to answer the following two questions with our experimental results:

  1. Is there any performance difference between the single-server and two-server case?
  2. As compared to NSS-HA Large configuration, XL configuration doubles its storage bandwidth. What’s the performance gain if we compare their aggregate throughputs?

Please note that the results presented in the following figures are aggregate throughput. For example, the “4 concurrent test case” means that there are four clients concurrently accessing each file system, and the result is the aggregate throughput of the four-client read or write. That is, for NSS-HA Large configuration, there are four client threads accessing one file system, while there are eight client threads in total concurrently accessing the two file systems for NSS-HA XL configuration (four client threads per file system).

Figure 2 shows the test bed used in our evaluation. The test bed consists of 64 PowerEdge R410 servers in an HPC cluster used as I/O clients. The clients can access the two NFS servers via either 10GbE or InfiniBand. The NFS servers are setup in an HA-cluster and provide access to the backend PowerVault storage arrays.

The performance results based on IB

Let’s first answer our first question: Is there any performance difference between the single-server and two-server cases?

Answer: Yes, there are significant read and write performance differences between the single-server and two-server cases with the increase of the number of concurrent clients.

From figure 3 and figure 4, it is obvious that when there are a small number of concurrent clients per file system (less than or equal to 4), the peak performance between the two cases are not significantly different (less than 20% for write, 10% for read). However, as the number of concurrent clients per file system increases, we observed a difference of almost 40% for writes and 60% for reads between the two cases with 32 concurrent clients.

It is reasonable to observe such differences. When there are a small number of clients accessing the storage system, even a single server can provide sufficient system resources including CPU, memory, and I/O bandwidth to process the incoming I/O requests. Thus, the performance difference between the single-server and two-server cases is small. As the number of concurrent clients increases, the system resources in a single server become insufficient to host so many concurrent I/O requests, but the aggregate system resources for the two-server case are still sufficient to host those concurrent I/O requests. As a result, a significant performance difference is observed between the two cases.

Now, let’s answer our second question: what’s the performance gain if we compare the aggregate throughput between NSS-HA XL and large configuration?

Answer: For the two-server case, the aggregate read and write throughputs are close to twice of the performance of NSS-HA large configuration, as the system resource is always sufficient in that case. While for the single-server case, the aggregate read and write throughputs are close to twice of the performance of NSS-HA Large configuration when there are a small number of concurrent clients. As the number of concurrent clients increases, the performance gain is smaller than the two-server case due to the limited -system resources in a single server.

Figure 3

Figure 4

The performance results based on 10gbE

We observed the similar performance behavior as the case of IB, although the similar behavior is due to a different reason: the network bandwidth is the limiting factor for the single-server case. Figure 5 and 6 show the performance results.

Figure 5

Figure 6

Failover functionality Evaluation

To verify the failover functionality for NSS-HA XL configuration, we conducted different tests with various initial states and different failures. Table 1 lists our results.



Table 1. Failover test results
Failure type Initial Status Result
Kernel panic on a single server One server hosts two NFS services. Two NFS services successfully failover to the other server.
Each server hosts a NFS service. The NFS service on the failed server successfully failover to the other server.
10GbE network connection failure on a single server One server hosts two NFS services. Two NFS services successfully failover to the other server.
Each server hosts a NFS service. The NFS service on the failed server successfully failover to the other server.
IB network connection failure on a single server One server hosts two NFS services. Two NFS services successfully failover to the other server.
Each server hosts a NFS service. The NFS service on the failed server successfully failover to the other server.
Heartbeat link failure on a single server One server hosts two NFS services. Two NFS services successfully failover to the other server.
Each server hosts a NFS service. The NFS service on the failed server successfully failover to the other server.
The failover time is less than 2 minutes, which matches the time observed in the NSS-HA L failover tests.

Furthermore, we also tested the failover functionality when there is client I/O accessing the file system. Our results are promising. The clients are able to complete the I/O operations after failover, although they cannot access the file system through the failed server during the failover period.


Note: there is a little difference between the NSS-HA Large configuration file and XL configuration file. We attach the XL configuration file below.


By Xin Chen and Garima Kochhar