Successfully Running ESXi from SD Card or USB – Part 2

In Part 1 of this blog, we discussed some items that need to be addressed to successfully run ESXi from an SD card or USB drive. Most specifically the syslog files, core dump, and VSAN traces files (if VSAN is enabled).

This post will discuss some options to address each one and various pros and cons to each method. Unfortunately there is no definitive answer. Since each infrastructure can and will be different it is nearly impossible for me, Dell, or VMware to say exactly what you should do. The intent of the information below is to help give you options on how to manage these files. These are by no means the only options.

How do we manage these files, so they are persistent?

  1. Syslog Files
    1. Every IT environment will have a different preferred way to manage syslog files. Some datacenters already have a central syslog server that ESXi can use to send its logs. Here are two additional solutions.
    2. Solution 1 – vSphere Syslog Collector:
      1. Available to be installed for Windows vCenter installations. For the vCenter appliance, the syslog collector service is enabled but will not register as an extension. To retrieve the files you have to login via SSH to the vCenter appliance as they will not be available from the vCenter client.
      2. This application is aptly named vSphere Syslog Collector.
  • The installation is included on the vCenter install disk and can co-exist with vCenter. For a large environment, use another VM as a standalone Syslog Collector, to avoid overtaxing the vCenter server.
  1. Pros and Cons:
    1. Pros:
      1. Free!
      2. Easy to install.
    2. Cons:
      1. Stores syslog files in a flat file system. This can make finding data difficult.
      2. No integration with other VMware tools (i.e. vRealize Operations)
      3. Although status can be seen from vCenter, not much less can be managed from vCenter.
    3. Solution 2 – vRealize Insight
      1. vRealize Insight is a full blown log management and analytic solution.
      2. Installation is done by deploying virtual appliances. The number of appliances depends on the ingestion rate. Since vRealize Insight can be used for all types of system logging the amount of data can be considerable.
  • Pros and Cons:
    1. Pros:
      1. Actual analytics for the logs that are collected.
      2. Can manage logs sent from vSphere, Windows, Linux, etc.
      3. Easy to manage.
      4. Integrates with vRealize Operations.
    2. Cons:
      1. Not free.
      2. High IO consumer, especially for storage.
    3. Core Dump Files
      1. Core Dump files are pretty easy to manage.
      2. As long as you are using at least a 4GB SD/USB drive, you are covered. The core dump will be saved to the 2.2GB partition reserved for the dump.
        1. ESXi doesn’t need 4GB but if you plan on running a system with considerable amount of memory (i.e. >256GB but <512GB) or VSAN, it is required. Note that systems with more than 512GB of RAM are required to use local storage.
      3. Another solution to manage core dumps is to setup vSphere Dump Collector similar to vSphere Syslog Collector.
        1. vSphere Dump Collector is a service that can coexist on your Windows based vCenter to collect dumps from hosts configured to send core dumps.
        2. Pros:
          1. Free!
          2. Allows for a central location for host diagnostic data.
          3. Will be able to hold the entire dump (no limitation on dump file size vs SD/USB).
          4. Available on both Windows based vCenter and vCenter appliances.
  • Cons:
    1. ESXi uses a UDP based process to copy the core dump files to the target location. UDP is unreliable as packets are sent without verification of delivery, so data could be missing and you will not know it.
  1. VSAN Trace Files
    1. There are only a couple of options to manage VSAN trace files. They cannot be sent to a syslog server.
    2. Solution 1 – Let them write to the default RAMDisk.
      1. Pros:
        1. Easy to setup (i.e. no setup!)
        2. VMware natively supports copying these files to the SD/USB drive during host failures (not total hardware failure though).
      2. Cons:
        1. Increases the memory usage of the host. This can be an issue on memory constrained systems.
        2. If the VSAN trace files are larger than the available space in the locker partition, not all will be copied during a host failure.
      3. Solution 2 – Send them to an NFS server.
        1. Pros:
          1. VSAN trace files are kept on a remote storage location so even in the event of a complete host hardware failure the trace files are available.
          2. Easy to setup.
          3. Reduces the load on the host’s memory.
        2. Cons:
          1. Requires an NFS server or storage location.
          2. Additional network traffic out of the host.
        3. Solution 3 – Write to a local VMFS data store.
          1. Pros:
            1. A good solution if no NFS server or storage is available.
          2. Cons:
            1. Wastes drive(s) and drive space that could be used for VSAN drives.
            2. Requires the host to have a second disk controller as VSAN drives need to be connected to a dedicated controller. This can be tricky in some configs – for example, if your preferred platform is the R630, it does not support multiple controllers.
            3. Additional cost for drives just to support VSAN trace files.

Remember, not every solution above will work in your environment.  But I do strongly advise doing something to protect, at the very least, the core dumps and VSAN trace files.  These are two key items that either VMware support will require to help resolve issues that may come up.  With the available free options it is cheap insurance for what could be a terrible support/troubleshooting session.

Look for the third and final blog in this series where I will show you how to configure some of the infrastructure discussed above.