Last week the Dell HPC engineering team presented a poster at the IEEE Cluster 2009 conference in New Orleans describing iSCSI I/O subsystems for HPC. Commonly used in Enterprise computing, the HPC community has been slow to adopt iSCSI storage for clusters. This paper describes two use cases for iSCSI based I/O subsystems for HPC clusters.

The first approach -- called an indirect SAN -- has servers re-exporting iSCSI storage via a traditional parallel file system. With the second approach -- called a direct SAN -- the compute nodes attach directly to a clustered file system on the iSCSI storage via software iSCSI initiators. The direct (top) and indirect (bottom) topologies are depicted in the diagrams below.


The study finds that the indirect approach adds scalability at the expense of increased cost and performance overhead. With the indirect SAN approach, I/O bandwidth can be improved or maintained as the cluster grows by adding additional data movers. Furthermore, depending on the characteristics of the underlying file system and iSCSI storage, the indirect approach can improve performance by coalescing client requests. However, careful thought must be given to the ratio between data movers, clients, and storage arrays.

The scalability of the direct approach is limited by the capabilities of the underlying storage. For example, the MD3000i is limited to 16 client nodes while the PS 6000 series iSCSI SAN can scale to hundreds of client nodes. The advantage of the direct appraoch is that it eliminates the cost and performance associated with the data movers. The direct approach allows customers to allocate more of their budget to the workhorses that solve the problems.


Of course this is not ground breaking scientific research, but the direct SAN approach is a novel idea that merits further exploration. Given that commodity, standards based hardware and open source software are the forces that drive the widespread adoption of HPC clusters, iSCSI seems like a natural fit. iSCSI can provide flexible, block level access to shared storage without eroding a cluster's favorable economics.

The paper accompanying the poster includes design recommendations for both approaches along with performance and scalability data. It will be published in the conference proceedings and at IEEE Xplore. I'll write about my impressions of the Cluster 2009 conference next week.