Friday, November 30, 2012

Graceful shutdown of an ESXi 5.1 host and guest VMs (free edition) using the shell/command line/scripting (UPS friendly)

Update 2/11/2013: A much easier method for doing this has been documented in this blog post.  Thanks to reader Everett for the suggestion!

Update 2/7/13:  A shell script that does what this post describes has been posted at github.  Enjoy!

On a single ESXi 5.1 host (INCLUDING the free edition), I have been able to gracefully shutdown, poweroff or reboot the host and guest VMs using the commands documented below from the ESXi 5.1 shell.

You may want to do this in response to an uninterruptible power supply (UPS) power failure event trigger.  In that case, you will need to install at least one guest VM (consider the VMware Virtual Management Appliance) that can run your UPS' software or Linux's Network UPS Tools (NUT).  You do a USB or serial passthrough of your UPS (for locally connected ones).  When the UPS software indicates a shutdown is required, you can run a shell script on the guest VM that SSH's into the ESXi 5.1 host and then runs a shell script on the host or some variation of these commands directly.

For example: a UPS trigger fires on the VMA appliance, runs remote-shutdown.sh which does SSH into your host ESXi 5.1 as root (consider using key authentication so you don't need to store the password) and runs the a shell script on the host (host-shutdown.sh) which you store on a datastore local to the host ESXi server.

Or you might just want to shut things down or do other maintenance via the shell/command line which these commands allow you to do.

The two command-line tools used here are vim-cmd and esxcli.

If you type vim-cmd, or vim-cmd <namespace> the tool has pretty good command-line help for figuring out what it can do - and that is quite a bit!

NOTE:  I have not seen this method documented elsewhere and so you must assume this method is not officially supported by VMware - but it seems to work fine (and it may be able to be be improved on as well)! 

Command List/Sequence:

1) list all vms

~ # vim-cmd vmsvc/getallvms

2) gracefully shutdown a vm (uses the VM's "world id") - you can also use power.off, power.reboot, power.suspend, etc.
 
~ # vim-cmd vmsvc/power.shutdown <VM/"world id" from step 1>

3) enter maintenance mode (immediately with no delay, this can only be done if ALL guest VMs have been shut down)

~ # esxcli system maintenanceMode set -e true -t 0

4) shutdown the ESXi host server

~ # esxcli system shutdown poweroff -d 10 -r "Shell initiated system shutdown"

5) try to exit maintenance mode real quick before shutdown!

~ # esxcli system maintenanceMode set -e false -t 0

If step # 5 does not succeed, your system will reboot in maintenance mode and you will have to manually take the system out of maintenance mode and restart your guest VMs.  
 
These commands can be built into a simple shell script that you can then deploy on the ESXi host server itself.  I have written one such script, and you can download it from GitHub.

Download esxidown (via github)

There may be more information available on this VMware forums post (11/30/2012).

64 comments:

  1. Hi! Ive been looking for something like this for agesm did you ever make a script?

    ReplyDelete
    Replies
    1. In fact, I did - I have updated the blog post with a link to the source on github. Enjoy!

      Delete
    2. dongtam
      mu private
      tim phong tro
      http://nhatroso.com/
      nhac san cuc manh
      tổng đài tư vấn luật
      http://dichvu.tuvanphapluattructuyen.com/
      văn phòng luật
      tổng đài tư vấn pháp luật
      thành lập công ty
      http://we-cooking.com/
      chém gió
      trung tâm ngoại ngữ
      - Có tất cả sáu người, năm người thuộc viễn cổ hổ tộc, một người thuộc về Hắc Ám Thần Điện, thực lực hình như là ngũ tinh Đấu Thánh, còn viễn cổ hổ tộc thì có ba lục tinh Đấu Tôn và một nhất tinh Đấu Thánh, còn có một ngũ tinh Đấu Thánh.


      Nhất tinh Đấu Thánh của viễn cổ hổ tộc kia sợ hãi mà nói.

      - Các ngươi đã nói nói xong sẽ thả ta đi, người của Tứ Đại Nhân Tộc nói xong phải giữ lời.

      Nhất tinh Đấu Thánh nói xong với Nhạc Thành thì lập tức cất tiếng.

      - Thả ngươi? dĩ nhiên, dĩ nhiên ta sẽ tiễn ngươi một đoạn đường.

      Nhạc Thành đứng dậy, hắn thu liễm khí tức toàn thân, viễn cổ hổ tộc và Hắc Ám Thần Điện có hai ngũ tinh Đấu Thánh tiến tới, Phá Quân sơn không cách nào chống lại, nếu Nhạc Cốc cùng với Khiếu Thiên Hổ không thể kịp thời khiêu chiến thì tuyệt đối là hậu quả rất lớn.

      Delete
  2. what are the advantages of doing this instead of just ssh a poweroff or halt command?

    ReplyDelete
    Replies
    1. The script tries to shutdown the guest VMs cleanly before shutting down the host. As far as the actual commands used to power off or halt the host itself, you can do that however you like. /sbin/shutdown.sh and /sbin/poweroff might work fine as Everett suggests below. But poweroff/halt or even /sbin/shutdown.sh will not shut down guest VMs under any circumstance that I am aware of.

      Delete
    2. In the tests I've done on the free ESXi 5.0 and 5.1, the poweroff command will shutdown any vm's that are set to autostart before powering off the host, but it won't shutdown any that aren't set to autostart.

      Delete
  3. Hi, an easier way might be to execute the built-in ESXi shutdown script. Just execute the /sbin/shutdown.sh script and configure ESXi to automatically shutdown the guest via VMWare Guest Tools.

    ReplyDelete
    Replies
    1. I forgot one additional thing. After the shutdown script you'll have to execute /sbin/poweroff and then wait several minutes for the host to power down.

      Delete
    2. I should add that shutdown.sh didn't work for me unless the host was in maintenance mode on 5.1 (free) but it may have worked on older versions (ESXi 4 for example).

      Delete
    3. Interesting, I'm also running 5.1 free; I wonder how my configuration differs from yours. Do you have your host configured to shutdown guests automatically?

      Delete
    4. I don't have access to the system that needed this anymore but I think it had something to do with clustering. I guess entering maintenance mode first is a "safer" host shutdown especially in a cluster.

      I also had problems with ESXi 5.0 U1 - that may have been the root cause of the problem, but I no longer have that either. I know they had a lot of problems with /sbin/vmware-autostart.sh in that version (which is the same script called by shutdown.sh). For example, this thread-> http://communities.vmware.com/message/2068088

      There is also this terribly misleading VMware KB document regarding RAID controller cache on shutdown which states that one should enter maintenance mode (when in vSphere!!!) before shutting down the host for ESXi 5.1: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1013193

      Your method does work, I was really surprised when it didn't work before for me. Thanks for bringing up the solution - I will update the post with your feedback. It really should be the way you described.

      I re-downloaded ESXi 5.1 (free) and put two CentOS minimal guest VMs on it with VMware tools installed, set to auto-shutdown with the host.

      Issuing /sbin/shutdown.sh and /sbin/poweroff shut down the guest VMs and host without issue. Looks like whatever I encountered is no longer an issue.

      Thank you for your feedback and suggestion, Everett!!! You clearly went out of your way to share the information. You are right also!

      Delete
    5. I added a follow-up post here-> http://www.nojokeit.com/2013/02/shut-down-esxi-51-guest-vms-and-host.html. Thanks again for your input!

      Delete
  4. you can create a nice list of all (!) vmsvc commands with this short script:
    ---------------
    # make a single-columned file:
    vim-cmd vmsvc|grep -v "^Commands available" | awk '{ print $1 }' >file1.txt
    vim-cmd vmsvc|grep -v "^Commands available" | awk '{ print $2 }' >>file1.txt
    # log every commandname in the the file:
    echo "set -x" >file2.txt
    # insert vim-cmd help in front of each command:
    cat file1.txt | sed 's#^#vim-cmd help vmsvc/#g' >>file2.txt

    # Create the whole help-file
    sh file2.txt >file3.txt 2>&1

    # make it nicer:
    cat file3.txt | sed -e "s/^+/====>/g" >helpfile.txt

    ---------------

    /foo # head helpfile.txt
    ====> vim-cmd help vmsvc/acquiremksticket
    Usage: acquiremksticket vmid

    Acquire mks ticket.

    ====> vim-cmd help vmsvc/acquireticket
    Usage: acquireticket vmid ticketType

    Acquire a virtual machine ticket.

    /foo #


    oh.. there are problems with special characters => http://paste.debian.net/236229/

    ReplyDelete
  5. hi
    thank you for the very useful script.
    I have a questions
    I'm trying to create a cron job to schedule a shutdown for the ESXI every day at 16:00 PM:

    first I edited the /var/spool/cron/crontabs/root file

    00 16 * * * /sbin/shutdown.sh

    but it's not working

    help please

    ReplyDelete
    Replies
    1. I think what you may be running into is that changes made to the the crontab file on ESXi are only temporarily and are lost when the machine is rebooted.

      What you might do is a workaround - make sure that the line is added to the crontab file on every reboot - here's a helpful link that explains it -> http://www.somethingsomewhere.net/sheduled-shutdown-esxi-v5/

      By the way, if you haven't seen it, there is an easier way to shutdown the VMs explained in another one of my posts -> http://www.nojokeit.com/2013/02/shut-down-esxi-51-guest-vms-and-host.html

      Delete
    2. Thank you for your reply.
      I tried a lot of solutions but none of them working.
      I don't think there is a problem with the script or commands.
      because when I execute:
      # /sbin/shutdown.sh
      # /sbin/poweroff
      the ESXI shutdown correctly.
      but when I add a cron job with the same commands nothing happen.

      Delete
    3. Hi

      Add this script in /etc/rc.local

      /bin/echo "0 16 * * * /sbin/poweroff" >> /var/spool/cron/crontabs/root

      This script would stop the Esxi host everyday at 16:00.

      On Esxi Version 5.1 to 5.5 , editing /etc/rc.local is no longer valid and you need to use
      /etc/rc.local.d/local.sh
      instead.

      Delete
  6. Valuable information. Lucky me I found your site by
    chance, and I am stunned why this twist of fate didn't came about
    earlier! I bookmarked it.

    Look at my website :: locksmith plano

    ReplyDelete
  7. Using the cheap website design drag & drop file upload functionality and the convenient right-click context menus, you can effortlessly manage everything related to your website.

    ReplyDelete
  8. Looks like what I need, but I am a novice at all this and need some simple help

    SERVERIDS=$(vim-cmd vmsvc/getallvms | sed -e '1d' -e 's/ \[.*$//' | awk '$1 ~ /^[0-9]+$/ {print $1}'

    This line has me stumped - My IDs a 1,2,3 & 4 - where do I put them - "# specify IDs separated by a space " but I dont see it in this line as to where they go.

    Sorry, but I am a noob at this

    ReplyDelete
  9. I am happy when reading your blog with updated information! thanks alot and hope that you will post more site that are related to this site. air conditioning repair denver

    ReplyDelete
  10. If you know how to review best resume writing service, you know how to choose the best one for you among the hundreds of presented online. These writing services could be very helpful with your resume or any other ordered paper.

    ReplyDelete
  11. Atari Breakout Online: there are a lot of colorful blocks which you have to break by using a red ball and a paddle. How to play run 3 game: You always have to run in space. Use the up arrow to jump, the left arrow and the right arrow to control the direction. free online mahjong, also spelled majiang, mah jongg, and numerous other variants, is a game that originated in China. It is commonly played by four players

    ReplyDelete
  12. Great! Thanks for sharing the information. That is very helpful for increasing my knowledge in this field.
    Subway Surfers | Run 3 | Click Jogos



    Great! Thanks for sharing the information. That is very helpful for increasing my knowledge in this field.
    Temple Run 2

    ReplyDelete
  13. Go find someone to love and love is always an arduous journey. In every way, we will meet each of these types of people with different ways of love. And after all, what we collect is sometimes not completely original as desired. Some people will be happy after going through many turbulent times was able to clench his hand injured person.
    run 2 , b612 app , slither.io

    ReplyDelete
  14. I found a lot of information here to create this actually best for all newbie here. Thank you for this information.
    red ball 4 | ninjago game | strike force kitty 2 | red ball | ninjago games

    ReplyDelete
  15. I would like more information about this, because it is very nice. Thanks for sharing. Game return man play the football game for free. The game return man 3 you like it? I like play game as Stick RPG and game Potty Racers 3

    ReplyDelete
  16. When the unfortunate fate of life enlightened us arrange encounters, surely someone has and will appear in our lives will bring a certain lofty task. Each man stepped forward and left will leave the great lesson, that if not in the days of youth, the fear is that we have not been able to remember and lessons deep within.

    Leer más: http://www.salseandoenlacocina.com/2011/01/gracias.html#ixzz4bSQk9aJr
    Under Creative Commons License: Attribution Non-Commercial No Derivatives
    whatsapp baixar
    Square Quick
    Geometry Dash
    Baixar Musicas
    Snapchat

    ReplyDelete
  17. Life becomes more interesting and wonderful when you share your memorable moments with friends and family through unique photographs. You can create your own unique style impressed with image editing software. And after hours of work stress you can also

    dream league soccer
    subway surfer
    clash royale
    go launcher
    banana kong

    ReplyDelete
  18. We are covering most of the cities and still giving you the best services within very short period with top notch satisfaction. We can trace us 24/7 and you are always welcome.

    ReplyDelete
  19. You have even figured out how to make it reasonable and straightforward to peruse. You have some authentic work ability. Much thankful to you.

    SEO Interview questions

    ReplyDelete
  20. I found a lot of information here to create this actually best for all newbie here. stick-rpg2.com

    ReplyDelete
  21. Very helpful advice in this particular post! It’s the little changes that make the largest changes. Thanks for sharing!

    http://word-cookies-answers.com

    ReplyDelete
  22. You can create your own unique style impressed with image editing software. And after hours of work stress you can also www.whatsappbaixargratis.com.br

    ReplyDelete
  23. On the off chance that crediting cash would pull away money related weight away, this sort of advance can without much of a stretch be the solution to your necessities.

    ReplyDelete
  24. The way that there is an alternative at all is the thing that makes these credits positive. At times individuals simply require an open way to turn their accounts around and a payday credit does the trap.

    ReplyDelete
  25. In the event that you can't stand to fork over the required funds, rub together as much as you can. This will bring down the intrigue expenses the following term around.

    ReplyDelete
  26. Thank your site! information your share is useful to me! it's great that I known this site!

    ReplyDelete
  27. I found a lot of information here to create this actually best for all newbie here. Thank you for this information.
    www.appbaixar.com

    ReplyDelete
  28. Thank you very much for the information you shared, it’s all I’ve been looking for

    ReplyDelete
  29. I can feel that the articles contained in this blog is so interesting. I also get a variety informasin, thanks. geometry-dash.net

    ReplyDelete
  30. I was very impressed by this post, this site has always been pleasant news. Thank you very much for such an interesting post. Keep working, great job! In my free time, I like play game: facebooklite.com.br. What about you?

    ReplyDelete
  31. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. www.facebookbaixar.com.br

    ReplyDelete
  32. I really like your blog! Continue to write more! Very interesting! swordsandsouls.net

    ReplyDelete
  33. I enjoyed over read your blog post. Your blog have nice information, I got good ideas from this amazing blog. I am always searching like this type blog post. I hope I will see again. fireboy-watergirl.com

    ReplyDelete
  34. Welcome, your blog is great. It's very interesting and meaningful, it helps me have more knowledge about life. dr-driving.com

    ReplyDelete
  35. Alot of blogs I see these days don't really provide anything that I'm interested in, but I'm most definately interested in this one. Just thought that I would post and let you know. Nice! thank you so much! Thank you for sharing.
    b612az.com

    ReplyDelete
  36. Thank you very much for the information you shared, it’s all I’ve been looking for
    www.happywheels-2.com

    ReplyDelete
  37. I enjoyed over read your blog post. Your blog have nice information, I got good ideas from this amazing blog. I hope I will see again. www.whatsappbaixargratis.net

    ReplyDelete
  38. Your article is very interesting. I think this article has a lot of information needed, looking forward to your new posts. Get permission to share:
    returnman3game.com

    ReplyDelete
  39. Your article has a lot of good information. It proves you understand very well about this issue, I agree completely on your article.
    http://carriagereviews.com/best-4-channel-amps

    ReplyDelete
  40. I appreciate you and hopping for some more informative posts. thanks for share
    run3.run

    ReplyDelete
  41. I read deeply your article, the points you have mentioned in this article are useful potova

    ReplyDelete
  42. I am looking for my memories through the stories, the narrative of people. I feel it is difficult but I will try.
    povaup

    ReplyDelete
  43. Just thought that I would post and let you know. Nice! thank you so much! pocaauto

    ReplyDelete
  44. I love all the posts, I really enjoyed, I would like more information about this, because it is very nice., Thanks for sharing.

    virginia tees

    ReplyDelete
  45. Thank you, so Much admin have a nice day :) you information about weight loss is very easy and simple.
    pocafashion

    ReplyDelete
  46. Your post is interesting. Hope you have more useful posts like this
    http://vex3.games

    ReplyDelete
  47. Payday advances are substantially less demanding to get and in less time than different credits. In the wake of applying for a payday advance on the web, you will get a phone call from the payday bank.

    ReplyDelete
  48. We should have a recap first. Payday advances are here and now, quick and bother free advances that are the perfect answer for any startling deficiency of cash that may emerge in your life.

    ReplyDelete
  49. I was very impressed by this post, this site has always been pleasant news. Thank you very much for such an interesting post. Keep working, great job! In my free time, I like play game: imgtaram. What about you?

    ReplyDelete
  50. I have read your article, the information you give is very interesting.
    instagram viewer

    ReplyDelete
  51. I love all the posts, I really enjoyed, I would like more information about this, because it is very nice., Thanks for sharing.
    badicecream3.co

    ReplyDelete