One of the initial steps in setting up a cluster is to configure the BIOS and the IPMI address on every cluster node. Optimal BIOS configuration can improve application performance as well as save power. IPMI configuration allows the cluster to be managed remotely.

This blog post describes a set of scripts that can be used to set HPCC specific BIOS options and configure the IPMI controller on each node. The Cluster Administrator just needs to have these scripts saved on to a single directory on the master node and run the main.sh script. The main.sh script will take certain inputs from the Cluster Administrator and in turn run bmc_front_end.sh, values.sh and compute_ipmitool.sh scripts in serial.

The scripts use the Linux “ipmitool” command to configure the iDRAC (IPMI compliant Dell remote access controller) on PowerEdge R and M series servers and BMC (Baseboard management controller) on C6100. The “syscfg” utility from Dell Deployment Toolkit (DTK) is used to configure the Dell BIOS. To install syscfg on all compute nodes, follow the instructions in this blog post < http://www.delltechcenter.com/page/Changing+BIOS+settings+with+syscfg+from+the+DTK> and then proceed with the steps below. However if using a Dell supported cluster package, you would find that syscfg is already bundled along with Dell.


The scripts assume an NFS share on the master that can be accessed by all compute nodes. They also use a parallel shell to issue commands to all hosts. The example scripts below use the “tentakel” utility that is packaged with ClusterCorp Rocks+ but this can be replaced with any parallel shell like “pdsh”.

These scripts have been verified on Dell 11th generation PowerEdge servers like the PowerEdge R610.

At a very high level, the flow of the script is as below:
1. Take the required inputs from the Cluster Administrator:
a. First two octets of the BMC monitoring IP {$bmc_2_octets}
b. Subnet mask of the BMC monitoring network {$bmc_mask}
c. Gateway for the BMC monitoring network {$bmc_gw}
2. Create an additional eth0:0 interface on the master node with an IP address comprising of $bmc_2_octets and last 2 octets of the eth0 IP address.
3. Create a configuration template script that sets HPCC specific BIOS options and also configures the BMC of the compute nodes.
4. Copy the configuration template script to the nodes and execute the script locally on each compute node.
5. The configuration script will set the HPCC specific BIOS options and configure the BMC parameters on each node. A log file is created on each compute node and in case of any error the Cluster Administrator can use the file to troubleshoot any issues. After configuring BIOS and BMC on each compute nodes, a reboot is initiated for these changes to take effect.


A simple 2 node cluster with this solution is illustrated below:

Let us consider an example where on the master node IP assigned to eth0 is 172.20.0.1 and to eth1 is 10.10.0.1. Interface eth0 is the provisioning or private network for the cluster. Interface eth1 is the public interface. If the Cluster Administrator chooses to create a BMC monitoring network with the input of 192.168.0.0/255.255.255.0/192.168.0.1, the following sequence of steps is executed by the scripts:
1. Cluster Administrator provides the required inputs and in turn the scripts set values for the variables bmc_2_octets, bmc_mask and bmc_gw.

2. The script now creates an additional BMC monitoring interface on the master node with the device name eth0:0 as follows:
a. The IP address of eth0:0 is constructed using the last 2 octets of eth0 (In our example eth0 has an IP of 172.20.0.1, hence 0.1 will be taken) and the first 2 octets ($bmc_2_octets) as provided by the cluster adminstrator.
b. The subnet mask and the gateway are the same as bmc_mask and bmc_gw.
The eth0:0 will have the following parameters: IP Address=192.168.0.1, Subnet Mask=255.255.255.0 and Gateway=192.168.0.1

3. Now the main script creates a configuration template script which is executed on each compute node individually. This configuration template script has 4 sections in it:
Section 1: Set the values for the BMC subnet mask and gateway as per the input provided by the Cluster Administrator.
Section 2: Provide the syscfg commands to set the HPCC specific BIOS settings.
Section 3: Determine the IP parameters that have to be assigned to the local BMC on the respective nodes. For a compute node compute-0-1 with an eth0 IP of 172.20.0.253/255.255.0.0, the BMC IP should be 192.168.0.253/255.255.255.0.
Section 4: Provide the ipmitool commands to set the IP parameters to the local BMC on the respective nodes.

4. The main script will first copy the configuration template script created in the previous step to an NFS mounted shared location on the master node, for example, /share/apps. This folder on the master node is mounted across all the compute nodes. The main script now copies this configuration template script from /share/apps to /root location on all the compute nodes.

5. Finally the main script uses a parallel shell command like tentakel to execute the configuration template script locally on each compute node. Each compute node is rebooted for the changes made to BIOS and the BMC to take effect.


I have provided the scripts used below. In this implementation, note that master node does not configure the BMC of any compute nodes. The configuration of the BMC on each node is done locally using the configuration template script saved to it.

main.sh

#! /bin/bash
#
# main.sh
# Copyright (C) 2009-2010 Dell, Inc.
# by Calvin Jacob <calvin_jacob at dell.com>
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
#echo "Create the BMC Monitoring Interface on your Front-End"
echo " "
sh ./bmc_front_end.sh
sh ./values.sh
echo " "
echo "Creating BMC Monitoring Interface on Front-End with a device name eth0:0"
echo "#! /bin/bash" >> bmc_ip.sh
echo " "
cat bmc_ip.sh values compute_ipmitool.sh >> bmc_network.sh
cp bmc_network.sh /share/apps/
service network restart >> /dev/null
echo "Configuring BMC and setting HPCC recommended BIOS options on compute nodes"
echo " "
rm -f bmc.* bmc_ip.sh bmc_network.sh values
tentakel "cp /share/apps/bmc_network.sh /root" >> /dev/null
tentakel "chown root:root /root/bmc_network.sh" >> /dev/null
tentakel "dos2unix /root/bmc_network.sh" >> /dev/null
tentakel "rm -f /root/bmc_bios.log" >> /dev/null
tentakel "sh /root/bmc_network.sh > /root/bmc_bios.log" >> /dev/null
tentakel "rm -f /root/ip_file" >> /dev/null
echo "You can access the BMC of any node in this cluster using the first 2 octets of BMC monitoring IP of front end and the last 2 octets of the system IP address of the node. In case you are not able to access BMC of any of the nodes, please refer to /root/bmc_bios.log on the respective nodes. You may reconfigure the BMC of a particular node by running /root/bmc_network.sh on the respective nodes."
bmc_front_end.sh
#! /bin/bash
#
# bmc_front_end.sh
# Copyright (C) 2009-2010 Dell, Inc.
# by Calvin Jacob <calvin_jacob at dell.com>
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
#read -p "First 2 octets of the BMC Monitoring IP (example: 192.168):" bmc_2_octets
echo $bmc_2_octets >> bmc.file
cp bmc.file bmc.net
echo . >> bmc.file
cat /etc/sysconfig/network-scripts/ifcfg-eth0 | grep IPADDR | cut -f2 -d'=' | cut -f2 | cut -f3 -d '.' >> bmc.file
echo . >> bmc.file
cat /etc/sysconfig/network-scripts/ifcfg-eth0 | grep IPADDR | cut -f2 -d'=' | cut -f2 | cut -f3 -d '.' >> bmc.file
export bmc_ip_addr=`sed -e :a -e '$!N; s/\n//; ta' bmc.file`
rm -f /etc/sysconfig/network-scripts/ifcfg-eth0:0
echo "DEVICE=eth0:0" >> /etc/sysconfig/network-scripts/ifcfg-eth0:0
echo "IPADDR=$bmc_ip_addr" >> /etc/sysconfig/network-scripts/ifcfg-eth0:0
read -p "Subnet Mask of the BMC Monitoring IP (example: 255.255.255.0):" bmc_mask
read -p "Gateway of the BMC Monitoring IP (example: 192.168.0.1):" bmc_gw
echo "NETMASK=$bmc_mask" >> /etc/sysconfig/network-scripts/ifcfg-eth0:0
echo "BOOTPROTO=static" >> /etc/sysconfig/network-scripts/ifcfg-eth0:0
echo "GATEWAY=$bmc_gw" >> /etc/sysconfig/network-scripts/ifcfg-eth0:0
echo "ONBOOT=yes" >> /etc/sysconfig/network-scripts/ifcfg-eth0:0
echo $bmc_mask > bmc.mask
echo $bmc_gw > bmc.gw
rm -f bmc.file
ifdown eth0:0
ifup eth0:0

values.sh
#! /bin/bash
#
# values.sh
# Copyright (C) 2009-2010 Dell, Inc.
# by Calvin Jacob <calvin_jacob at dell.com>
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
#
echo "export bmc_net_mask=`cat bmc.mask`" >> values
echo "export bmc_gw=`cat bmc.gw`" >> values

compute_ipmitool.sh
#! /bin/bash
#
# compute_ipmitool.sh: Script that does something
# Copyright (C) 2009-2010 Dell, Inc.
# by Calvin Jacob <calvin_jacob at dell.com>
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
#rm -f ip_file
dell_model=`/usr/sbin/dmidecode | /bin/grep –i poweredge | /usr/bin/awk '{print $3}'`
if [ "$dell_model" == "PowerEdge" ]; then
/opt/dell/toolkit/bin/syscfg --virtualization=disable
/opt/dell/toolkit/bin/syscfg --cpuc1e=enable
/opt/dell/toolkit/bin/syscfg --logicproc=disable
/opt/dell/toolkit/bin/syscfg --turbomode=enable
/opt/dell/toolkit/bin/syscfg --cstates=enable
/opt/dell/toolkit/bin/syscfg --serialcomm=on
/opt/dell/toolkit/bin/syscfg --conboot=enable
/opt/dell/toolkit/bin/syscfg --memintleave=disable
/opt/dell/toolkit/bin/syscfg power --profile=maxperformance --setuppwdoverride
else
echo “BIOS parameters not set on this server.”
fi
/etc/init.d/ipmi stop
/etc/init.d/ipmi start
/usr/bin/ipmitool lan set 1 ipsrc static
/usr/bin/ipmitool user set password 2 calvin
/usr/bin/ipmitool lan set 1 access on
echo $bmc_gw | cut -f1 -d'.' >> /root/ip_file
echo . >> /root/ip_file
echo $bmc_gw | cut -f2 -d'.' >> /root/ip_file
echo . >> /root/ip_file
cat /etc/sysconfig/network-scripts/ifcfg-eth0 | grep IPADDR | cut -f2 -d'=' | cut -f2 | cut -f3 -d '.' >> /root/ip_file
echo . >> /root/ip_file
cat /etc/sysconfig/network-scripts/ifcfg-eth0 | grep IPADDR | cut -f2 -d'=' | cut -f2 | cut -f4 -d '.' >> /root/ip_file
export bmc_ip_addr=`sed -e :a -e '$!N; s/\n//; ta' /root/ip_file`
/usr/bin/ipmitool lan set 1 ipaddr $bmc_ip_addr
/usr/bin/ipmitool lan set 1 netmask $bmc_net_mask
/usr/bin/ipmitool lan set 1 defgw ipaddr $bmc_gw
rm -f ip_file
/usr/bin/reboot