Dell Community
High Performance Computing Blogs

High Performance Computing

High Performance Computing
A discussion venue for all things high performance computing (HPC), supercomputing, and the technologies that enables scientific research and discovery.
  • HPC Power & Cooling: Getting IT Contained

    Today, there is a lot of buzz about implementing containment in data centers. IT organizations are grappling with the challenge of getting enough cool air to their equipment while also keeping their operating costs under control. Containment provides one method to reduce the amount of air conditioning that is being provisioned into the data center by as much as 50%. It’s interesting to look at the wide range of solutions being devised to help address this challenge. The options offered tend to fall into three categories: room-level, row-level, or rack-level. Many of these are active solutions that require either external power or plumbing.

    1. Room-level: You can design your entire data center room to contain cold air under the floor and hot air above the ceiling, and provide the necessary ducting to route the air accordingly. This is great for an all-new facility. However, if you’ve got an existing space to work with, your options for redesigning the room may be constrained.
    2. Row-level: Aisle containment, either in the hot aisle or the cold aisle, uses hanging partitions or solid panels attached to the top and sides of the racks. These offerings are available from multiple vendors and typically require customization around a fixed number of racks. Depending on the configuration, facilities such as fire suppression systems may have to be modified for use within the contained area.
    3. Rack-level: Containment can be done on the hot side or cold side. Hot side solutions include chimney ducts, which may need to be connected to the ceiling or other duct work, and rear door coils, which require water or refrigerant for cooling the hot exhaust. One option for cold side containment is the Energy Smart rack, which seals to the floor to passively direct air into an integrated chamber to distribute it equally to all installed equipment. It can be easily positioned over a ventilated grate in a data center using raised-floor cooling, and doesn’t require any modification to the room’s facilities.

    HPC Power & Cooling Getting IT Contained - The Dell TechCenter

    One key factor for managing operating costs is to implement best practices in your data center deployment, such as hot/cold aisle configuration and using blanking panels in any unused rack space. Taking the basic steps for temperature separation and airflow paths will help to improve efficiency in the data center.

    While there is no single solution for every installation, it is certainly possible to identify what the best option is for a particular implementation, taking into consideration the tradeoffs. For example, in a data center with raised-floor cooling, rack-level containment can provide better support for higher density deployments than aisle containment.

    However, this can induce pressure on the server, so the server level implications need to be understood. If you need help determining the best containment solution for your data center, Dell Services can perform an assessment at your facility. If a rack-level solution is right for you, the Energy Smart rack provides integrated containment, natural high density capability, and greater room distribution control.

    Feel free to share your insights on this topic. What are the cooling challenges you are dealing with? Which containment solution fits best for your data center environment? What are the key features that are most interesting or beneficial to you? We’d enjoy hearing back from you.

    Joyce RuffJoyce Ruff
    Joyce Ruff is a product marketing manager for the Data Center Infrastructure team within the Dell Enterprise Product Group, and is responsible for the rack enclosures and accessories that complement Dell server solutions.

  • HPC Power and Cooling: BTUs/hr – Part 5

    John Fragalla
    Dell's John Fragalla

    Referencing Wikipedia, British Thermal Units (BTUs), and when used as a measurement of power, is expressed as BTUs per hour (BTUs/hr), is the amount of heat required to raise the temperate of one pound of liquid by 1 degree Fahrenheit. In a HPC system, BTUs/hr is the measurement of heat the system produces to aid in sizing the cooling solution in a data center. Correctly sizing the cooling solution based on BTUs/hr will ensure adequate cooling is providin to the system through traditional Computer Room Air Conditioning (CRAC) units and other supplemental cooling solutions, such as In-Row Chillers or Rear Cooling Doors. BTUs/hr is derived from the following formula:

    BTUs/hr = Watts * 3.41

    In the above formula, Watts is the peak measured power consumption of the HPC system running Linpack, as discussed in Part 2 of my series of HPC Power and Cooling blogs. The heat produced by the system is directly related to the peak measured watts when running Linpack.

    In part 6 of my blog, the next topic to discuss for a HPC system is Tons of Cooling, and how it relates to BTU/hr.

    John Fragalla, Dell HPC Senior Technologist

  • HPC Power and Cooling: CFMs – Part 4

    CFMs, cubic feet per minute, for a data center is the amount of air being delivered to the cold aisles. CFMs are also discussed as the amount of air produced by fans internal to components of a HPC system. If the CFMs being produced by the system exceed the maximum CFMs a data center can deliver to the cold aisles, hot air may be circulating into the cold aisles, can create hot spots in a rack or in certain sections of the data center, increase the likely hood of component failures, and possibly exceed the temperature limits of components causing power outage and downtime.

    One key formula I use when I want to calculate CFMs for particular components of a HPC system, such as a compute node, is the following:

    CFMs = (3412*W/1000) / (1.085 * ∆T)

    In the above formula, W represents the peak measured power consumption in watts when stressing the components with a particular benchmark, such as Linpack discussed in Part 2 of my blog, and ∆T is the change in temperature between the air entering the system components and air existing the system components. When the change in temperature is large, typically seen when the inlet temperature is low, the CFMs are lower. But if the change in temperature is small, typically seen if the inlet temperature is high, than the resulting CFMs is higher because the fan speed increases to remove the hot air quickly from the components. Typical measured change of temperature I personally seen at customer sites is anywhere from 20F to 30F, and can be as high as 40F to 50F, but this depends on each customer’s data center, the inlet temperature, and how well the hot aisle is contained to avoid hot air leaks into the cold aisle.

    Another general rule of thumb I also have used, and has worked well, is 100 CFMs for every 1 kilowatt of power measured.

    In part 5 of my blog, the next topic to discuss for a HPC system is British Thermal Units per hour (BTUs/h), which is a measurement of heat produced.

  • HPC Power and Cooling: Amps – Part 3

    John Fragalla
    Dell's John Fragala

    Amps of a HPC System is derived from the equation Wattage divided by Voltage (A = W/V). The Wattage, or just Watts, is usually the highest power consumption measured when running Linpack, as discussed in Part 2 of this blog. The voltage depends on the customer’s data center, but typical voltage used in the United States (US) is 208V. Amps is a critical measurement needed for a HPC system because it helps size the appropriate rack Power Distributions Units (PDUs) that is compatible with a customer’s data center.

    Rack PDUs come in different configurations, ranging from single phase power, to three phase power, and different amps per phase, such as 30A to 60A. Each HPC component, such as the compute nodes, switches, storage, has different circuit types, such as C13 or C19. When racking HPC components in a rack, one must find the appropriate rack PDUs to support the correct type of circuits to power up each of the HPC components. The decision between single phase or three phase power, and the amps per phase, is typically dictated by what is configured and support in each of the customer’s data center.

    After deciding on the correct rack PDU with the correct circuit type and type of power supported in the data center, another aspect to consider on the rack PDUs are breakers protecting a branch of circuits. Typically, these breakers are 20A, and in some cases in the US, they are de-rated by 80% to 16A due to UL specification. On the rack PDUs, 16A to 20A is the maximum amount of amps that breaker can support before tripping the breaker. The HPC component amps can be plugged into each branch of circuits without exceed 16A or 20A, depending on the breaker capacity and if they are de-rated.

    In some cases, depending on the amps per rack, one might need multiple rack PDUs per rack to balance the load across circuit breakers and phases to ensure, under peak load when running Linpack, no breakers are exceeding the maximum specification.

    In Part 4 of this blog, the next important topic to discuss for a HPC system is Cubic Feet per Minute (CFM), which is a measurement of air flow a HPC system produces from component fans.

    Read HPC Power and Cooling: Power Consumption - Part 1
    Read HPC Power and Cooling: Power Consumption - Part 2

    -- John Fragalla