Friday, November 30, 2012

Graceful shutdown of an ESXi 5.1 host and guest VMs (free edition) using the shell/command line/scripting (UPS friendly)

Update 2/11/2013: A much easier method for doing this has been documented in this blog post.  Thanks to reader Everett for the suggestion!

Update 2/7/13:  A shell script that does what this post describes has been posted at github.  Enjoy!

On a single ESXi 5.1 host (INCLUDING the free edition), I have been able to gracefully shutdown, poweroff or reboot the host and guest VMs using the commands documented below from the ESXi 5.1 shell.

You may want to do this in response to an uninterruptible power supply (UPS) power failure event trigger.  In that case, you will need to install at least one guest VM (consider the VMware Virtual Management Appliance) that can run your UPS' software or Linux's Network UPS Tools (NUT).  You do a USB or serial passthrough of your UPS (for locally connected ones).  When the UPS software indicates a shutdown is required, you can run a shell script on the guest VM that SSH's into the ESXi 5.1 host and then runs a shell script on the host or some variation of these commands directly.

For example: a UPS trigger fires on the VMA appliance, runs remote-shutdown.sh which does SSH into your host ESXi 5.1 as root (consider using key authentication so you don't need to store the password) and runs the a shell script on the host (host-shutdown.sh) which you store on a datastore local to the host ESXi server.

Or you might just want to shut things down or do other maintenance via the shell/command line which these commands allow you to do.

The two command-line tools used here are vim-cmd and esxcli.

If you type vim-cmd, or vim-cmd <namespace> the tool has pretty good command-line help for figuring out what it can do - and that is quite a bit!

NOTE:  I have not seen this method documented elsewhere and so you must assume this method is not officially supported by VMware - but it seems to work fine (and it may be able to be be improved on as well)! 

Command List/Sequence:

1) list all vms

~ # vim-cmd vmsvc/getallvms

2) gracefully shutdown a vm (uses the VM's "world id") - you can also use power.off, power.reboot, power.suspend, etc.
 
~ # vim-cmd vmsvc/power.shutdown <VM/"world id" from step 1>

3) enter maintenance mode (immediately with no delay, this can only be done if ALL guest VMs have been shut down)

~ # esxcli system maintenanceMode set -e true -t 0

4) shutdown the ESXi host server

~ # esxcli system shutdown poweroff -d 10 -r "Shell initiated system shutdown"

5) try to exit maintenance mode real quick before shutdown!

~ # esxcli system maintenanceMode set -e false -t 0

If step # 5 does not succeed, your system will reboot in maintenance mode and you will have to manually take the system out of maintenance mode and restart your guest VMs.  
 
These commands can be built into a simple shell script that you can then deploy on the ESXi host server itself.  I have written one such script, and you can download it from GitHub.

Download esxidown (via github)

There may be more information available on this VMware forums post (11/30/2012).

18 comments:

  1. Hi! Ive been looking for something like this for agesm did you ever make a script?

    ReplyDelete
    Replies
    1. In fact, I did - I have updated the blog post with a link to the source on github. Enjoy!

      Delete
  2. what are the advantages of doing this instead of just ssh a poweroff or halt command?

    ReplyDelete
    Replies
    1. The script tries to shutdown the guest VMs cleanly before shutting down the host. As far as the actual commands used to power off or halt the host itself, you can do that however you like. /sbin/shutdown.sh and /sbin/poweroff might work fine as Everett suggests below. But poweroff/halt or even /sbin/shutdown.sh will not shut down guest VMs under any circumstance that I am aware of.

      Delete
    2. In the tests I've done on the free ESXi 5.0 and 5.1, the poweroff command will shutdown any vm's that are set to autostart before powering off the host, but it won't shutdown any that aren't set to autostart.

      Delete
  3. Hi, an easier way might be to execute the built-in ESXi shutdown script. Just execute the /sbin/shutdown.sh script and configure ESXi to automatically shutdown the guest via VMWare Guest Tools.

    ReplyDelete
    Replies
    1. I forgot one additional thing. After the shutdown script you'll have to execute /sbin/poweroff and then wait several minutes for the host to power down.

      Delete
    2. I should add that shutdown.sh didn't work for me unless the host was in maintenance mode on 5.1 (free) but it may have worked on older versions (ESXi 4 for example).

      Delete
    3. Interesting, I'm also running 5.1 free; I wonder how my configuration differs from yours. Do you have your host configured to shutdown guests automatically?

      Delete
    4. I don't have access to the system that needed this anymore but I think it had something to do with clustering. I guess entering maintenance mode first is a "safer" host shutdown especially in a cluster.

      I also had problems with ESXi 5.0 U1 - that may have been the root cause of the problem, but I no longer have that either. I know they had a lot of problems with /sbin/vmware-autostart.sh in that version (which is the same script called by shutdown.sh). For example, this thread-> http://communities.vmware.com/message/2068088

      There is also this terribly misleading VMware KB document regarding RAID controller cache on shutdown which states that one should enter maintenance mode (when in vSphere!!!) before shutting down the host for ESXi 5.1: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1013193

      Your method does work, I was really surprised when it didn't work before for me. Thanks for bringing up the solution - I will update the post with your feedback. It really should be the way you described.

      I re-downloaded ESXi 5.1 (free) and put two CentOS minimal guest VMs on it with VMware tools installed, set to auto-shutdown with the host.

      Issuing /sbin/shutdown.sh and /sbin/poweroff shut down the guest VMs and host without issue. Looks like whatever I encountered is no longer an issue.

      Thank you for your feedback and suggestion, Everett!!! You clearly went out of your way to share the information. You are right also!

      Delete
    5. I added a follow-up post here-> http://www.nojokeit.com/2013/02/shut-down-esxi-51-guest-vms-and-host.html. Thanks again for your input!

      Delete
  4. you can create a nice list of all (!) vmsvc commands with this short script:
    ---------------
    # make a single-columned file:
    vim-cmd vmsvc|grep -v "^Commands available" | awk '{ print $1 }' >file1.txt
    vim-cmd vmsvc|grep -v "^Commands available" | awk '{ print $2 }' >>file1.txt
    # log every commandname in the the file:
    echo "set -x" >file2.txt
    # insert vim-cmd help in front of each command:
    cat file1.txt | sed 's#^#vim-cmd help vmsvc/#g' >>file2.txt

    # Create the whole help-file
    sh file2.txt >file3.txt 2>&1

    # make it nicer:
    cat file3.txt | sed -e "s/^+/====>/g" >helpfile.txt

    ---------------

    /foo # head helpfile.txt
    ====> vim-cmd help vmsvc/acquiremksticket
    Usage: acquiremksticket vmid

    Acquire mks ticket.

    ====> vim-cmd help vmsvc/acquireticket
    Usage: acquireticket vmid ticketType

    Acquire a virtual machine ticket.

    /foo #


    oh.. there are problems with special characters => http://paste.debian.net/236229/

    ReplyDelete
  5. hi
    thank you for the very useful script.
    I have a questions
    I'm trying to create a cron job to schedule a shutdown for the ESXI every day at 16:00 PM:

    first I edited the /var/spool/cron/crontabs/root file

    00 16 * * * /sbin/shutdown.sh

    but it's not working

    help please

    ReplyDelete
    Replies
    1. I think what you may be running into is that changes made to the the crontab file on ESXi are only temporarily and are lost when the machine is rebooted.

      What you might do is a workaround - make sure that the line is added to the crontab file on every reboot - here's a helpful link that explains it -> http://www.somethingsomewhere.net/sheduled-shutdown-esxi-v5/

      By the way, if you haven't seen it, there is an easier way to shutdown the VMs explained in another one of my posts -> http://www.nojokeit.com/2013/02/shut-down-esxi-51-guest-vms-and-host.html

      Delete
    2. Thank you for your reply.
      I tried a lot of solutions but none of them working.
      I don't think there is a problem with the script or commands.
      because when I execute:
      # /sbin/shutdown.sh
      # /sbin/poweroff
      the ESXI shutdown correctly.
      but when I add a cron job with the same commands nothing happen.

      Delete
    3. Hi

      Add this script in /etc/rc.local

      /bin/echo "0 16 * * * /sbin/poweroff" >> /var/spool/cron/crontabs/root

      This script would stop the Esxi host everyday at 16:00.

      On Esxi Version 5.1 to 5.5 , editing /etc/rc.local is no longer valid and you need to use
      /etc/rc.local.d/local.sh
      instead.

      Delete
  6. Valuable information. Lucky me I found your site by
    chance, and I am stunned why this twist of fate didn't came about
    earlier! I bookmarked it.

    Look at my website :: locksmith plano

    ReplyDelete
  7. Using the cheap website design drag & drop file upload functionality and the convenient right-click context menus, you can effortlessly manage everything related to your website.

    ReplyDelete