Monday, February 11, 2013

Shut down ESXi 5.1 guest VMs and the host (free edition) via SSH - the easy way!

Thanks go out to reader Everett for pointing out an easier way to gracefully shutdown guest VMs and the host on a VMware ESXi 5.1 (free) server. 

This is much easier than the method described in the previous post I made on the subject, although that method may have been necessary at the time due to issues with VMware 5.0 U1, or still may be necessary in special cases (clustering, etc.).  The previous method also gives you a tiny bit more control, but it is not supported.

You may want to gracefully shut down your guest VMs and host ESXi 5.1 server via SSH, for example, on the triggering of a UPS power outage event or something similar.

The method is as follows:

1) Install VMware Tools in all guest VMs.

2) Make sure each guest VM is setup to perform the shutdown action "Guest Shutdown" (or you could also use a suspend, if you wanted to) in the virtual host settings "Virtual Machine Startup and Shutdown" section.

3) The following two commands, run in sequence, will shutdown the properly configured guest VMs and the host server also:

/sbin/ && /sbin/poweroff

These commands can be run in sequence via an SSH connection from another system (for example, a batch file and plink on Windows, on a machine running a UPS).  The poweroff will only run if the script runs successfully.

4) That's it!

Thanks Everett!

Friday, November 30, 2012

Graceful shutdown of an ESXi 5.1 host and guest VMs (free edition) using the shell/command line/scripting (UPS friendly)

Update 2/11/2013: A much easier method for doing this has been documented in this blog post.  Thanks to reader Everett for the suggestion!

Update 2/7/13:  A shell script that does what this post describes has been posted at github.  Enjoy!

On a single ESXi 5.1 host (INCLUDING the free edition), I have been able to gracefully shutdown, poweroff or reboot the host and guest VMs using the commands documented below from the ESXi 5.1 shell.

You may want to do this in response to an uninterruptible power supply (UPS) power failure event trigger.  In that case, you will need to install at least one guest VM (consider the VMware Virtual Management Appliance) that can run your UPS' software or Linux's Network UPS Tools (NUT).  You do a USB or serial passthrough of your UPS (for locally connected ones).  When the UPS software indicates a shutdown is required, you can run a shell script on the guest VM that SSH's into the ESXi 5.1 host and then runs a shell script on the host or some variation of these commands directly.

For example: a UPS trigger fires on the VMA appliance, runs which does SSH into your host ESXi 5.1 as root (consider using key authentication so you don't need to store the password) and runs the a shell script on the host ( which you store on a datastore local to the host ESXi server.

Or you might just want to shut things down or do other maintenance via the shell/command line which these commands allow you to do.

The two command-line tools used here are vim-cmd and esxcli.

If you type vim-cmd, or vim-cmd <namespace> the tool has pretty good command-line help for figuring out what it can do - and that is quite a bit!

NOTE:  I have not seen this method documented elsewhere and so you must assume this method is not officially supported by VMware - but it seems to work fine (and it may be able to be be improved on as well)! 

Command List/Sequence:

1) list all vms

~ # vim-cmd vmsvc/getallvms

2) gracefully shutdown a vm (uses the VM's "world id") - you can also use, power.reboot, power.suspend, etc.
~ # vim-cmd vmsvc/power.shutdown <VM/"world id" from step 1>

3) enter maintenance mode (immediately with no delay, this can only be done if ALL guest VMs have been shut down)

~ # esxcli system maintenanceMode set -e true -t 0

4) shutdown the ESXi host server

~ # esxcli system shutdown poweroff -d 10 -r "Shell initiated system shutdown"

5) try to exit maintenance mode real quick before shutdown!

~ # esxcli system maintenanceMode set -e false -t 0

If step # 5 does not succeed, your system will reboot in maintenance mode and you will have to manually take the system out of maintenance mode and restart your guest VMs.  
These commands can be built into a simple shell script that you can then deploy on the ESXi host server itself.  I have written one such script, and you can download it from GitHub.

Download esxidown (via github)

There may be more information available on this VMware forums post (11/30/2012).

Thursday, September 6, 2012

Google Chrome pages not loading, pages appear gray

Update 11/27/2012 7:25PM: 

This is completely unconfirmed, but, on the user's desktop that had this problem, I discovered that the user who had the problem was infected with a previously UNKNOWN virus (stored in two randomly named dlls) which was submitted to Microsoft and VirusTotal for classification.  A few weeks later it was classified as the following severe threat/virus - "TrojanSpy:Win64/Ursnif.AK" - by Microsoft.  Here's the entry for that on the Microsoft website and on VirusTotal.

Once this threat was removed and Chrome reinstalled and updated to the latest version, the problem went away.  Not sure exactly how the two were related but, for this user, Chrome stopped working immediately the day after two suspicious virus related DLLs (based on their timestamp) were installed on the user's desktop.

Scans with the latest up-to-date version of Microsoft's Security Essentials caught the virus (and hopefully other anti-virus vendors have now implemented signatures for it as well).

Try an updated anti-virus scan and see if resolves your issue!

Update 9/6/2012 11:08AM: The solution below kind of works, but it is clearly only a temporary fix.  If you desperately need to use Chrome, you can just double click on the old_chrome.exe and it seems to work, however, if you use the regular "chrome.exe" it may eventually have a problem again.  Waiting on an official Google fix - see this post on the Google product forums.

I recently had a user who had Google Chrome, and when they opened it up, no pages were loading, and all pages (including configuration settings) appeared gray, and nothing at all would display.  The user's Chrome version is 21.0.1180.89 m and it looks like it was deployed 9/4/2012 at 3:19 AM (during the night).

The gray pages look like this:


I located the folder in which Chrome is installed.  In this case (Windows 7 64-bit), it was:

C:\Users\<your username>\AppData\Local\Google\Chrome\Application

where <your username> is the Windows user account you used to log on.


Although it is odd and may not work for everyone, I was able to run the program "C:\Users\<your username>\AppData\Local\Google\Chrome\Application\old_chrome.exe", and it loaded up a previous version of Chrome which loaded as it was normally expected.

Navigate using Windows Explorer in Windows 7 to:


Run "old_chrome" just once and then close the program and run Chrome as you normally do.
After loading the old_chrome.exe (presumably a back-up created by Google during an update just in case something breaks) once, I closed Chrome and re-opened it via the user's regular Chrome link on the desktop and it opened like normal.  I checked the version number and it still reports the new version number, so clearly opening the old_chrome.exe reset some setting or cleared something - and it seems to have fixed the problem.

I'm not sure completely why it fixed the problem (for that, we will have to wait and see what Google says regarding the issue), but it did.

Here is a related post ("Page not loading in Chrome") on the Google product forums that might help you if the fix above doesn't solve the problem for you.

Looks like a little Google bug.  Oops!

Wednesday, August 22, 2012

The Curse of "Being the IT Guy/Gal"

This is a truth that my fellow "IT Guys/Gals" can most likely identify with:

Whether you spend all day neck deep in desktops, servers, viruses, programming code, networking equipment or have your head "in the cloud", it doesn't really matter - when you get home at night (assuming you get to be at home at night, and are not working the night shift), you really don't want to mess with your own personal IT stuff.

It's a pain.  A delightful pain that I want to enjoy, but a pain nonetheless.  You spend all day doing one type of thing (or 1,000 similar types of things) and get home and get to do it... some more.  For that matter, we often have to do it, just to keep up our skills and keep learning!  It's a whole kind of pleasure/pain kind of deal and I don't like where this line of thought is heading!! o_O

I find it especially true for folks like me whose hobby became their job.  Hey, I'm grateful I have a job.. and damn lucky my hobby fit the bill - don't get me wrong.  That said, if you happen to be the local IT hero or MacGyver programmer of your office, where your day is spent doing anything from fixing printers, to writing shell scripts in Linux, to supporting legacy code, answering that support call (or 100s of them) or writing reams of new code - you know exactly what I'm talking about.

It's kind of like the doctor who doesn't (and often shouldn't) see his/her family members as patients.  While he could (and often does), there are moral, ethical and practical reasons why family members of a physician might need to have their own doctor, and not the one(s) that happen to live under the same roof.  And the saying "physician heal thyself" didn't come about for no reason!

Remember the saying "No, I will not fix your computer" - it may save your life, although it may not win you friends nor influence.  Even though I know, when your significant other or your mom/dad/brother/cousin's uncle's best friend's dog asks, you're going to feel guilty, and probably help them fix it/upgrade it/configure it anyway.  

Maybe better said, even if you love technology and want to mess with it 24/7, eventually, it will mess with you and when you feel that intense need to take a break and do something else - DO IT!

After all, if you don't, the machines win.  And we can't have that happen, can we?  It didn't go too well for John Connor.

Actually... envisioning my router with glowing blue LEDs as a T2... oh brother.. we're already there.  Haha, pulled the power cable - die T2 die!!  Oh crap, maybe I should plug that backbone connection back in.  I felt the power and now I feel the pain!

Heed the call fellow IT warriors, for constant IT work at home and on the job is a surefire path to burnout!  Remember to have some fun once in a while!

Now to try to take my own advice, and put down the keyboard and mouse for you know, ten minutes.  At least until the next server alert or upgrade or status bar appears.

Monday, July 23, 2012

ESXi 5.0 Auto-Start Broken - Fix/Patch Released!!

If you are using VMWare ESXi 5.0.0 U1 (the free version), there has been a bug with the virtual machine auto-start feature for quite some time, where machines would not start automatically if they were configured to do so.  Up until now, the only fix was to revert back to the old version (5.0.0).

It appears that a patch has been released - see this VMWare blog post for more information.  Here's the direct link to the patch on the VMWare website:

Here is a method of installing the patch via the CLI.

I applied the patch against an ESXi 5.0.0 U1 server in my lab by uploading it using vSphere Client to the main datastore, SSH'ing into the machine, and then running the following command:

esxcli software vib install --depot=/vmfs/volumes/datastore1/

I'm hoping they will roll this patch into the next major release... no idea when that comes out though.

Thursday, May 17, 2012

Should I buy Facebook stock? Why is FB valuable?

Should I buy Facebook stock?  The better question is, why is Facebook valuable?

On the eve of the Facebook (NASDAQ:FB) IPO, I felt compelled as a technologist to share an important observation about the company and its products:
The value of Facebook rests in identity, communication, sharing, and recording/making relationships between data about people, places, events and things.

Facebook has done what no other organization in the entire world has managed to do, and that is to catalog the identity (usually at least a name, photograph, maybe a hometown) of 800 million people.

Even such projects as OpenID, explicitly designed to try solve the problem of giving you one unique identifier to use at websites around the world, have never taken hold to the extent that Facebook has.

Using the simplest search tools, and the relationships you have with your contacts/friends (and the power of 6 degrees of separation), there is a very likely chance that you can locate and identify nearly any person you know, who is on Facebook, and that they can locate you.

When you step back, this is a problem that information technology/systems companies have been trying to solve for years.  It is a problem that tons of organizations (including governments) have spent billions of dollars trying to solve.  And it is a problem solved by the book of "faces"... Facebook.

With a fairly accurate "identity" to represent 800 million people, suddenly, data analysis of those people becomes a lot easier.  We are all just nodes on the network.  Think about it - driver's license number, social security number (SSN), government ID number, etc. - these are all unique identifiers that are often a contract between you and a government.  Some are private and not meant to be shared.

They are all made somewhat extraneous if one can verify identity using Facebook.

Your Facebook page is a public way of identifying yourself, to a large number of people and organizations.  This is the reason for the advice - watch what you say on Facebook, you never know who will see it and how long it will be around for!

The IPO is set to price around $38 a share and raise approximately $16 billion dollars at that price.

Think about what $16 billion can do in making your life better through ancillary services that surround Facebook, but making use of this core identity feature.

Add to it, the fact that every website you see these days, every news article, everything on the web, has a giant "LIKE" button next to it.  Everything is personalized to suit what these systems think your tastes are (also known as a "filter bubble").  What about all the other actions that could be tracked?

The like button is an action.  It is a communication tool, it is a sharing tool.  By clicking it, content from around the world gets tagged and stored in Facebook's giant "open graph" database.

I... like... something (already exists) or someone.
I... am in a relationship with... someone (already exists).
I... bought.. something.
I... talk to... someone.
I... went....  somewhere (think "Check-ins").
I... ate......  something.
I... made... something.
I... work... somewhere (job info).
I... have...  something.
I... saw...   something (why did they buy Instagram?).

Heck, with base assertions like these, predictive analytics about what I will do next, what I want, what I might buy - these are things that are eerily powerful.  If you fed all this into artificial intelligence tools, you'd probably be able to build a little model of my psyche (or at least, as much of it as Facebook knew).

If you are familiar with the popular game The Sims made by Will Wright you know how much data about mood, physical status, etc. the game tracks about your characters.  Imagine a Sims character generated from all this data that Facebook has collected about you - I wonder what that would look like!

If Facebook gets this right, $16 billion or more will help them turn open graph into the world's largest centralized repository of data about human activity of many different types that has ever existed, outside of perhaps technologies used by governments through intelligence gathering organizations.

Yes, there are privacy concerns.  Yes, the government wants to get at this data.

But there is more.  Just because the open graph database exists, does not mean that everything needs to exist within it.

Instead, Facebook could work with partner companies to build private, closed databases, where you are identified by your Facebook ID, but the Facebook application itself has NO access to the data inside these repositories.

Facebook wants to get into business organizations.  They are going to use much of this IPO money to try (or so I believe).

Facebook wants to be an integral part of your life, just "like a good neighbor, State Farm is there".  That's right, one of the most highly coveted marketing/advertising techniques companies love to use with people is - "we're with you from birth to death, at every step of the way".  Watch carefully and see how many television commercials use this technique for things as big as oil companies to General Electric.  Remember the timeline?  From birth to death?  Sound familiar?

Lately, there has been much discussion about electronic health records, and centralized healthcare systems, and health information exchange in the United States.

Do you think such data should be on Facebook?  How about your banking data?  How about any private data at all?

Of course not!  But, that's the thing people don't realize - the data doesn't HAVE to live inside of Facebook.  The data can be stored securely away, anywhere else, but access and identification of who you are could be done through Facebook.

Facebook is a platform to build on.  I believe this is what they will push the company forward with.  Use Facebook as a platform to build around.

Look at their recent announcement of the "App Center/Store".  Look at the tools that Facebook offers for building applications that live inside of Facebook?

There is value here!  It's crazy, scary and powerful.

Or is there value?  And is it crazy, scary, powerful?

One thing is for sure - they didn't get to where they are without having succeeding at implementing many major innovations and new ideas, and I would wager a guess that they will continue pushing hard and growing out however they can.

Tuesday, May 15, 2012

Zyxel Zywall - VPN issues, firmware updates, problems, review

Firmware updates for Zywall products

If you happen to have any Zyxel Zywall products (such as the USG 50, USG 200, etc.), keep your eyes out for firmware updates.  It is clear to me that they are consistently having to update their firmware, and there have been a lot of changes recently.  For the USG 200 product alone, there have been at least two firmware updates in only 3 months!

Zyxel Zywall USG 200 firmware updates -

All Zyxel support downloads - "Download Library" -

It would be nice if they sent out an email notification every time their was a new firmware release!

Nailed-up VPNs

Regarding setting up VPNs on Zywall USG products, if you have a problem where your VPN connections do not restore automatically after a reboot, you may consider activating the "nailed-up" option in the Advanced Settings section of the VPN Connection tab for that particular connection.

According to a knowledge base article on the Zyxel site (article 1633), "nailed-up" as applied to PPP connections means: 
"A nailed-up connection is always up regardless of there is traffic transmitted. The ZyXEL Device performs two actions when the nailed-up feature is enabled. First, connection idle timeout is disabled. Second, the ZyXEL Device will try to bring up the connection when turned on and whenever the connection is down. A nailed-up connection can be very expensive for some reasons. It is always a better idea not to enable a nailed-up connection unless the broadband service provider offers flat-rate service or you need a constant connection and the cost is not a concern. You can enable/disable WAN connection nail-up in SMT menu 11 or the web GUI."
I did not find useful documentation, but I believe the term "nailed up" as applies to VPN has the same meaning - so in case you want your VPN connections to dial automatically after a reboot, consider this setting!  So far, I have not seen any negative affect for having used it.   If you were paying for metered/limited bandwidth and leave your VPN "nailed up", though, you may have consequences from your connection being in use at all times, so do be careful.  That said, for most people, site-to-site VPN connections are meant to be up constantly, so it isn't a problem.

Quick review of Zyxel as a vendor

Overall, having worked with a lot of vendor's firewalls over the years (Sonicwall, Watchguard, Zyxel and Cisco to name a few), I have to say, the Zyxel stuff is affordable, but not entirely intuitive and somewhat roughly documented.  That said, their tech support seems pretty responsive and helpful, as long as you only need their help during business hours (8-5PM Pacific Time).
Phew, it's been a while since I had time to post any tips!