ESXi lost configuration on reboot

Servers

Information and ideas on Dell PowerEdge rack, tower and blade server solutions.

ESXi lost configuration on reboot

Not Answered This question is not answered

Hi,

 

Recently we've installed ESXi 4.1 Update 1 embedded version on the internal SD card (SanDisk 2GB) for a few Dell PowerEdge M610 blade servers using the ESXi recovery CD from Dell, also available for download from the VMware site:

http://downloads.vmware.com/d/details/dell_esxi_recoverycd/ZHcqYip3ZWJkKmV3

 

Yesterday we rebooted the servers to make sure everything would come back up.

To our surprise when ESXi come back up on each server, the whole configuration was gone: IP address, server name, password, everything was lost, the configuration was back as if we just installed ESXi.

 

Any ideas why that happened?

 

We did a proper reboot for ESXi: one of the servers was rebooted using vCenter, the others were rebooted using the ESXi DCUI using the F11 key.

 

Before the reboot, one of the ESXi server was up with its configuration for a few days and was managed through vCenter, the others were installed an hour or so before the reboot and weren't connected to vCenter yet: no matter how long they were up, all of them lost the configuration after the reboot.

 

And we've been unable to reproduce the issue so far, we've reconfigured ESXi on a couple of those servers today and rebooted them from the ESX DCUI: each of them came back online just fine this time, the configuration is still there.

 

We didn't reconfigure all of them, let me know if there is anything we can check on those still untouched after the reboot to tell how come they lost the configuration.

 

Thanks.

All Replies
  • I too have had this happen.

    Need to do some testing...

    Phil

  • Could you share some information about your configuration and your findings?

    Thanks.

  • Hey all,

     

    I have the same problem with a brand new t310. I did a reboot and all is gone: NIC settings, VMs deleted from the overview, license key deleted...

  • coffefan

    I have the same problem with a brand new t310. I did a reboot and all is gone: NIC settings, VMs deleted from the overview, license key deleted...

    Which brand/model of SD card are you using?

    Which edition/version of ESXi?

    Thanks.

  • Hey, 

    I have installed ESXi 4.1 Update 1 on mirrored HDDs (2x 250GB Dell HDDs). in the past 2 days I did a couple of reboots, no problems at all ( I did that in console view by hitting F12). 

    Today it was the first time I used the "maintenance mode" and rebooted the machine via the GUI. The server came up again but lost all settings. Ok, I reconfigured ESXi and all is fine again, but it is still strange... fortunately the VM is still there.

    Dave Mishchen posted here http://communities.vmware.com/thread/250892 to try the command to check if ESXi can save its config: all works fine.

    Cheers

  • It looks like there is an issue with the Dell ESXi ISO that can be reproduced very easily (even on a VM): 1) install ESXi and when installation is complete reboot 2) apply ESXi configuration: network, etc. 3) reboot with F11 4) configuration is lost.

    The issue seems to affect only first reboot, here is the explanation of the bug from a post by Dave Mishchenko on the VMware forum:

    oem.tgz on Hypervisor1 contains a tweak for MD36xxi support in the form of a VIB which means that /altbootbank being set as the active boot partition for the next reboot is correct.  They should have included a reboot command in  the customization. Since the host doesn't reboot  your changes get written to /bootbank either when you reboot or when the auto backup process runs (1 min past the hour).

    Is Dell planning to release a fixed ISO anytime soon?
    Is there any Dell email address to report the issue, in case they are not aware of it already?
    Thanks.

  • Hello, Dell is aware of this issue. It will be fixed and the new ISO will be released soon.  Will keep you posted in this thread.

    Thanks

  • Thanks, is there any ETA for the fixed ISO release?

    Thanks.

  • If you've actually used this Dell customized ISO (from Dell) and have a rather complex configuration, is there something you can do to prevent losing it on the first reboot?

    Member since 2003

  • Hello pzero, The issue is root caused. Now we are looking for the best possible fix. There is no ETA defined as of now. But we will have the images posted soon. Will keep you posted on the updates.

    Hello Dev Mgr,W.r.t the backup, VMware provides a way of backing up the data. We need to see if we can take backup of all the ESXI configuration data. Could you please elaborate on the specific scenarios when you say complex configuaration?

    As of now, we recommend the users to 'reboot' the system after first bootup of Dell Customized ESXi. From the second boot onwards, it's assured that the configuration changes will be persistent.

    Thank you,

    Krishnaprasad

  • I have setups with iSCSI, vMotion, multiple VLANs, etc. To recreate them would take 30 to 60 minutes per host.

     

    If there is a command I can run to tell ESXi to create altbootbank and then copy /bootbank to altbootbank?

    Member since 2003

  • Hello,

      There are 2  workarounds available as of now for the users who use the current ISO images posted @ Dell's support website.

    1.       Reboot the server after first bootup of Dell Customized ESXi image

    2.       Take backup of the configuration before rebooting the server after first boot. Here are the detailed steps for taking backup and restore.

    1.       Install the Dell Customized image posted @ the Dell Support website where the issue is seen

    2.       Booted into ESXi ( First boot )

    3.       Then perform the configuration changes like changing password, creating a vSwitch etc

    4.       Install VMware vSphere CLI which can be downloadable from VMware site --> ( http://downloads.vmware.com/d/details/vcli41/ZHcqYmRoaCpiZHdlZQ== )

    5.       Run command prompt in Windows and navigate to the directory "C:\Program Files (x86)\VMware\VMware vSphere CLI\bin"

    6.       Execute the command, vicfg-cfgbackup.pl --server < ESXi IP Address> --username root -s c:\backup.txt  where backup.txt will have all the configuration backup.

    7.       Reboot the system and boot into ESXi again

    8.       You can observe that all the configuration changes have been lost.

    9.       Connect the ESXi server from VI Client installed on a windows system

    10.   Navigate to Configuration --> Software --> Licensed Features tab

    11.   Note down the license key integrated into the host ( THis is mandatory because we are going to change the system's license to evaluation mode in next step )

    12.   Change the license to 'Evaluation Mode' by pressing edit button and selecting "Evaluation Mode"

    13.   Run command prompt in Windows and navigate to the directory "C:\Program Files (x86)\VMware\VMware vSphere CLI\bin"

    14.   Execute the command, vicfg-cfgbackup.pl --server < ESXi IP Address> --username root -l c:\backup.txt

    15.   Step 14 will ask the user to type yes on the command prompt to reboot the system . Type yes so that the host can be rebooted.

    16.   After reboot, you can see that all the configuration done on first boot is back.

    17.   Change the license key from 'Evaluation Mode' to the edition that Dell Provided, Navigate to Configuration --> Software --> Licensed Features tab

    18.   Click edit and select "Assign a new license key to this host" option and enter the License key as XXXXX-XXXXX-XXXXX-XXXXX-XXXXX

     

    Here are the set of configuration changes i have tried after first bootup and could see that all of them are recovered after taking backup and restore as above.

     

    1. Password changed to password

    2. IP Changed to 10.10.156.156

    3. Hostname changed to Testing-Customer-issue

    4. Modify Tech Support timeout changed to 720 secs

    5. Created vSwitch vSwitch1 with VM Network portgroup. Created vSwitch1 and VM Network2. Enabled VMotion and Management Traffic in vSwitch0.

    6. Change License ( to eval )

    Let me know if you have any questions.

    Thanks,

    Krishnaprasad

  • Hello,

       The image which contains the fix for the issue discussed in this thread will be available in Dell support Website early next week ( probably by Tuesday  - 22/3/2011 )

    Thanks, Krishnaprasad

  • Same thing happened to me last week, ESXi 4.1 update 1 Dell custom install cdX2 R710 with embedded SD cards. Ran fine for a week, then shutdown to move the equipment power-up and all the configuration was gone. Only took a few minutes to reconfigure, but this could be a big problem.

     

    Did you resolve and come to a conclusion? My customer is a "bit" concerned.

Page 1 of 2 (18 items) 12