Cluster implementations vary in the way the individual compute nodes boot. In the simplest case, the compute nodes each have their own local boot disk, with a bootstrap and operating system. In other implementations, the compute nodes have no local storage and they boot over an internal network, pointing to a common boot image stored on a boot node. There is also a hybrid implementation where compute nodes initially boot from a common boot image, and then finish booting on their own locally stored operating system. These latter two cases, where a common boot node is used, require a degree of fault tolerance in their design.

The use of a boot node eases the administrative burden because all compute nodes (which could easily be on the order of thousands) can use the same operating system image to boot. It becomes obvious that such an implementation would require the boot node to be operational most of the time. There are two things we can do to mitigate the risk of a boot node failure.

One obvious way to provide fault tolerance is to have multiple boot nodes running with the same image available. The compute nodes can then be configured to select a boot node in a random or round robin fashion. In this way, the failure of a single boot node will have little impact on the booting of compute nodes.

The second way to provide fault tolerance is a bit more subtle. If we configure the compute nodes to continue processing in the event of a boot node failure, we move the boot node failure risk to only those intervals when we are trying to boot a compute node. For example, as part of the compute node boot process, we could copy the OS image to a local compute node RAM disk, and boot the system from there. Typically, compute nodes have an uptime measured in months or years, so this scenario dramatically decreases the risk of a boot node failure.

To configure compute nodes to continue running in the absence of its boot node (as suggested above), we must configure the boot image such that dependencies on the boot node are removed at boot time. This involves moving critical portions of the operating system to local storage on the compute node.

Next Up… JOB SCHEDULING

-- Blake Gonzales

See other posts from this Blog series:
INTRO
SMP
CLUSTERED SYSTEMS
CLUSTERED SYSTEM INFRASTRUCTURE
POWER DISTRIBUTION
COOLING
MEMORY
LOGIN/HEAD NODES
COMPUTE NODES