C1E (Enhanced halt state) is a new BIOS option which is added to 11G Servers with the latest BIOS version 1.2.6. Since all other BIOS options (Turbo mode, C-states and various power management profiles) are already covered in the whitepaper “Optimal BIOS Settings for High Performance Computing (HPC) with PowerEdge 11G Servers” published by the HPCC Engineering team, I just wanted to use this space to talk about some of the details on C1E. I will also be covering the impact of C1E on performance/power consumption based on a number of HPC applications.

To begin with, I think it would be a good idea to talk about C-states. When C-states option is enabled in the BIOS, the individual processing cores on your server can transition to any of the below state depending on the work load:
  • C0: State in which instructions are being executed by the core.
  • C1: The core is active but no instructions are executed. The clock is gated thereby reducing the current drawn by the core.
  • C3: The core is inactive and the core caches are flushed.
  • C6: The core is inactive. The core caches are flushed and the power gates are used to reduce power consumption close to zero.
C1E is an enhancement to the C1 state and it is added as a separate option with the latest BIOS revision.
  • C1E: Enhanced Halt state – Both clock speed and voltage are reduced which results in improved power savings.
One important point to note here is that the C1E option can be set independent of the C-states option, ie C1E can still be enabled even if the C-states option is set to disabled. The BIOS default value is C1E enabled.

Now let’s get into the performance aspects and energy efficiency. Since the whitepaper already recommends T, C, Max (Turbo enabled, C-states enabled, and Maximum Performance power profile) for best performance, I just didn’t want to try out too many options here. The idea was to set Turbo mode-enabled and Max Performance for all the runs and toggle C-states and C1E settings for each run. Finally decided to lock down on four different combinations for this study:

  • T, Max (Turbo enabled, C-states disabled, Max Performance profile, C1E disabled)
  • T, C, Max (Turbo enabled, C-states enabled, Max Performance profile, C1E disabled)
  • T, Max, C1E (Turbo enabled, C-states disabled, Max Performance profile, C1E enabled)
  • T, C, Max, C1E (Turbo enabled, C-states enabled, Max Performance profile, C1E enabled)
The configuration used for the study is mentioned below: ·
  • 2 x PE R610 ·
  • Processor: Intel E5530 2.4 GHz (Turbo rating: 1/1/2/2). ·
  • BIOS Version: 1.2.6 (adds support for C1E). ·
  • 48 GB Memory (4 GB - 2 DPC 1067 MHz) Balanced configuration. ·
  • RHEL5 Update 3. ·
  • Intel C and Fortran compilers v11
  • Open MPI 1.3.2

Before getting into the application runs, the first thing I wanted to look into was the idle power consumption of the nodes under various BIOS settings. The nodes were kept idle for about half an hour with just the OS services running. The below graph gives a comparison of the idle power consumption. T-Max combination is taken as the base to which all the other settings are compared here:

Impact of C1E on PowerEdge 11G Servers -- HPBD 100909 - The Dell TechCenter

Two observations here:

  • With T-C-Max and T-C-Max-C1E, the idle power consumption is almost the same. C1E doesn’t contribute much here.
  • With just C1E enabled and C-states disabled, the nodes are still able to save ~12% power compared to T-Max.
PS: These values are read directly from the Power Management bus using Open Manage utilities and shows system board level power utilization.

Next objective was to see if C1E has any significant impact during application runs. Selected two applications here: CG, LU from the NAS parallel benchmark suite and executed on all the 8 cores of a single node.

For CG and LU, the performance and power consumption levels were almost the same with different BIOS settings combination as shown below. I have not added the LU graph here, as the pattern is very much similar to that of CG.

Impact of C1E on PowerEdge 11G Servers -- HPBD 100909 - The Dell TechCenter

The recommendation for HPCC is to keep C1E as enabled.

  • There doesn't seem to be any negative impact even for computation intensive workloads like CG and LU.
  • C1E can help in improved power savings during periods of inactivity in your cluster, even when C-states is disabled.