VMware Host Client uility: ESXi Embedded Host Client

The VMware Host Client is an HTML5-based client that is used to connect to and manage single ESXi host directly from web browser. It can be used to perform administrative tasks to manage host resources such as virtual machines, networking, and storage. The VMware Host Client can also be helpful for troubleshooting individual virtual machines or hosts if vCenter Server and the vSphere Client are unavailable.

The ESXi Host Client is part of ESXi 6.0 Update 2 now: Earlier it was available as a ESXi Embedded Host Client Fling only, so we have to install it before connecting to host via browser (supported for ESXi5.x and later).
The utility is bundled as a vib and can be installed on ESXi host via SSH without the need to reboot the host.
To use it on earlier versions of ESXi (prior to ESXi 6.0 Update 2), download and upload the Host Client VIB to any accessible datastore and then connect to host using putty and run following command,
#esxcli software vib install -d /vmfs/volumes/datastore1/esxui-signed-3623722.vib  

In case you downloaded the offline bundle, then

#esxcli software vib install -d /vmfs/volumes/datastore1/esxui-offline-bundle-5.x-3623722.zip

Once the VIB is installed, you simply just point your web browser to the following URL to access the host,

https://ESXi_HostName_or_IP/ui/ , For ESXi 5.5U2 and prior, and ESXi 6.0 hosts upgraded from any 5.5U2 or prior version, you will get a 503 error returned after visiting https://ESXi_HostName_or_IP/ui/. its a known issue and to resolve it, please remove the line starting with /ui from /etc/vmware/rhttpproxy/endpoints.conf and restart rhttpproxy with /etc/init.d/rhttpproxy restart
For ESXi 5.5U2 and prior hosts, you must ensure to append a trailing / (forward slash) to the URL after /ui. The URL must be https://ESXi_HostName_or_IP/ui/

In case if you want to remove the ESXi embedded host client from your ESXi host, then

#esxcli software vib remove -n esx-ui

In vSphere 6.0 Update 2, VMware continues to support the vSphere Client, but you now have the option of using the VMware Host Client instead of the vSphere Client to perform similar host-based operations.

VMware Host Client Features

The VMware Host Client 1.0.0 functions include, but are not limited to the following operations:

• Support for the latest hardware version.
• Basic virtualization operations, such as deploying, configuring, and editing virtual machines of various complexity, including console access to the virtual machines.
• Creating and managing network and datastore resources.
• Displaying current host and resource settings, including graphs of performance and utilization, and logs of host components to help troubleshooting.
• Advanced tuning of host level options to improve performance.

Full Release Notes is available here: VMware Host Client Release Notes

That's it... :)

VM has network connectivity but network card showing as disconnected, having red cross on it

In this post I will talk about this minor issue where, System has network connectivity but network card showing as disconnected, having red cross on it.

Earlier I had seen this problem and drafted a blog post about it but never posted until recently when one of my friend faced the same issue but couldn’t find its cause/fix.

Now coming to the point, you might have seen this where you deployed a VM from template or rebooted a system and when the VM came up, you are able to connect to it but its network card in notification area having red cross on it (appearing disconnected).

If you would check from connectivity point of view, you wouldn’t find anything wrong here however most of the time you would like to get rid of this red cross sign.

If you would further investigate about the issue, then you will find this in Network and Sharing Center,

Here to fix this, you just need to check the status of  “Network Location Awareness service”, in this case either its stopped or set to disabled. This service should be set to automatic and in started state. 

As this issue was caused by a windows services so I don't think its specific to VM only, you might see this issue on a physical Windows system as well.

Note: “Network Location Awareness service” is responsible to collect and stores configuration information for the network and notifies programs when this information is modified. If this service is stopped, configuration information might be unavailable. If this service is disabled, any services that explicitly depend on it will fail to start. 

That’s it… :)

Unexpected AD Account Lockouts When Logging On to Outlook

Today one of my friend asked me a AD account lockout related question, i.e. "there are two users and when they open outlook on their system, their AD accounts getting locked out automatically".

In such kind of cases, most of the time the issue is caused by a saved password, where user recently changed his account password but somehow outlook is not asking for new password and trying to connect using the old saved credentials instead.

To fix this account lockout issue, what one need to do is....remove the saved credentials from the system where your outlook account is configured.

To do so, Go to Start=> Control Panel => Credential Manager 

or open, Run => control keymgr.dll , this will also open the Credential Manager

You will find any saved credentials here,

Here Edit or Remove the saved outlook credentials and this will fix the AD account lockout issue. 

Alternatively we can do the same by using this method,

Click Start, Run => rundll32.exe keymgr.dll, KRShowKeyMgr (case sensitive)

 This will open the Stored User Names and Passwords window,

from here you can Edit/Remove or Add new credentials as well as Backup or restore the saved 
credentials.

Update: As per my discussion with a fellow Sys admin, Removing the credentials from Credentials Manager doesn't necessarily remove them from Key manager so sometimes we also need to check and remove the same from Stored User Names and Passwords. I am not sure if it's true as never seen this behaviour.

That's it... :)

Windows 7 VM auto suspend issue- reason and fix

Last day one of my colleague built a new VM with Windows 7 guest OS and after some time he noticed, once he logged off from this VM, its being automatically suspended after some time.
As he couldn't figure out the issue so asked me to take a look at this newly created VM to figure out why its being suspended automatically every now and then. When I checked, found the VM in suspended state, when taken a look at related task & events from vSphere client, couldn’t find anything in tasks however found suspend events under event tab.

However I couldn’t find any specific reason for this auto suspend from VMware end, then I logged in on to the Guest windows 7 OS  to check from inside the VM and started digging  System event logs and came across this,

Here it clearly mentioned, system was entered in sleep mode because it was ideal…..so what’s now…. Here we need to check from inside the guest, if the power plan for the Windows 7 is set to move the system to sleep if system is ideal for a specified time.

Now go to power options, Control Panel => Power Options => Click on Change when the computer sleeps

as you can see  here, this VM was set to enter in sleep mode if the system is ideal for 15 minutes.

Now what we need to do is,  set "Put the computer to sleep" to "Never" and you are done.

Note: I believe the same will be applicable for Windows 8 or any other client OS VM too.

This power plan setting is something that's by default set to never on server operating systems and that is why we never faced such issue with Server VMs.

That's it... :)

Uploading ESXi image to vSphere Update Manager failing with the error: Failed to import data

When it comes to ESXi host patching, I am a big fan of VMware Update Manager as it makes the upgrade process flawless. Using VUM, you just need to create upgrade baseline, download patches then attach baseline to host and finally stage/remediate the patches then update manger will initiate the patch installation and reboot the host once done.

Now coming to the point, we had few ESXi 5.0 hosts and wanted to upgrade them to ESXi5.1 update3 so I thought of upgrading them using VUM instead of command line.

As most of us know, to upgrade ESXi on an host using VUM, one first need to download ESXi iso image and import it to VUM ESXi image repository, here while trying to import the ESXi5.1 image to update manager5.5’s ESXi image repository, ESXi image import failed with below error,
"Failed to import data, The uploaded upgrade package cannot be used with VMware vSphere Update Manager"

I was aware about the fact that one need to upgrade the version of VUM before importing a newer image to update manager repository as the minimum version of VUM required should match the version of the ESXi image you are trying to import but here I was trying to upload ESXi5.1 image to VMware Upgrade Manager 5.5 and unable to do so. I tried with different ESXi5.1 images like Cisco or HP’s customized images as well as standard VMware ESXi5.1 image but the end result was same however when tried to import ESXi5.5 image to VUM, was able import it without any issue.

This is something that was very strange for me so started looking for the cause of this issue and my search end on the VMware KB articles# 2009812, if you would see the Note section of this KB, its clearly mentioned there,

“ You must use the same version of vSphere Update Manager to upgrade to the corresponding version of ESXi”

Later I checked with VMware support about this and they also confirmed the same.

Related KB article: 2097168

That’s it… :)

How to fix Sleep Mode issue on Windows 10

Last week I updated my laptop's OS from windows 7 to Windows 10 (64-bit) as well as installed all available patches.
Today while working on my laptop, left it for some time and when returned, noticed Sleep Mode having issues and the system didn't woke up from sleep then I had to press and hold the power button to power off/reboot the laptop. Then just to cross-check I closed the lid of laptop to put the OS in sleep mode and then again it it didn't woke up (yes there was no issue in power option config).

This is when I started looking for a fix and found a related discussion on HP Support forums 

however there its listed for Windows 8.1 but the same worked for my Windows 10 64-bit OS too.

This Sleep mode issue on Windows 10 is related to Intel Management Engine Interface (MEI) Driver and to resolve the issue do the following:

• Download version 9 or 10 version of the Intel Management Engine Interface (MEI) Driver form your respective System vendor site or directly from Intel itself (in case of mine, its a Lenovo 40-70 model laptop).

Look for the download in the "Driver-Chipset" category. If an Intel Management Engine Interface Driver with a version number starting in 9 or 10 is listed, download it (if you didn't find the Intel IME driver for Windows 10, then like me download the one listed for windows 8.1)

• If an MEI driver with a version number starting in 9 or 10 is NOT listed on system vendor site, then download the appropriate driver directly from Intel. [Version 9.5.24.1790 1.5M]
• Install version 9 or 10 of the Intel Management Engine Interface (MEI) Driver.

If you receive a dialog warning about replacing a newer version of the software, accept it. 

NOTE: You do NOT need to uninstall version 11 before installing version 9 or 10. The presence of some version of the driver is required in order to "upgrade" (or in this case downgrade) it.

If you already have a newer version of Intel IME installed then the setup will prompt for overwrite permission, click Yes.

• Once you Installed the Intel MEI driver version 9 or 10, Sleep Mode/Hibernate will start working again.

To avoid this issue in future, you may need to prevent the Intel MEI driver updating to version 11 via Windows update.To do so, 

• Download the Windows 10 "Show or Hide Updates" Troubleshooter Package

Change the Windows Update Services setting from Automatic to Manual.

 

•  Run the Windows 10 "Show or Hide Updates" Troubleshooter Package and hide updates to the Intel Management Engine Interface (MEI) Driver. (Doing so will block your system from automatically reinstalling or showing updates for version 11 of the driver.)

 Click Next,

• Now change the Windows Update Services setting from Manual back to Automatic and you are done.

That's it... :)

VMware vExpert 2016 Announced

VMware has announced the list of vExpert 2016…. I am very honored to be named a VMware vExpert again…..Congrats to all those named.

Here is the full list of vExpert 2016... http://blogs.vmware.com/vmtn/2016/02/vexpert-2016-award-announcement.html#comment-9968 

That's it... :)

Some useful storage related VMware Esxi commands

In this post I will talk about some useful storage related commands but before that I would like to mention about, getting help in Esxi local/remote shell or vCLI.

At any time to get help about any command or list all available namespace and options for a particular command we can use --help option.

We can use the --help option in same way for any other command.

Note: You can run these commands over local/remote shell or using vCLI. When using vCLI then you will also need to provide connection parameter and the command syntax would be as follows,

#esxcli --server ESXi_host_IP_or_Name --username noor --password noor2122 [<namespace> ...] <cmd> [cmd options]

                                                                 or 
#esxcli -s ESXi_host_IP_or_Name -u noor -p noor2122 [<namespace> ...] <cmd> [cmd options]

Now lets talk about some some useful storage related commands,

Command to view all LUNs presented to a host, 

#esxcfg-scsidevs -c  

And to check about a specific LUN,

#esxcfg-scsidevs -c | grep naa.id

To find the unique identifier of the LUN,  you may run this command:

# esxcfg-scsidevs -m

To find associated datastore using a LUN id

#esxcfg-scsidevs -m|grep naa.id

To get a list of RDM disks, you may run following command,

#find /vmfs/volumes/ -type f -name '*.vmdk' -size -1024k -exec grep -l '^createType=.*RawDeviceMap' {} \; > /Datastore123/rdmsluns.txt  This command will save the list of all RDM disk to a text file rdmluns.txt and save it to Datastore123.

Now Run following command to find the associated LUNs,             
#for i in `cat /tmp/rdmsluns.txt`; do vmkfstools -q $i; done
This command will give you the vml.id of rdm luns,

Now use following cmd to map vml.id to naa.id

#ls -luth /dev/disks/ |grep vml.id    in output of this command you will get LUN id/naa.id.

To mark an RDM device as perennially reserved:

#esxcli storage core device setconfig -d naa.id --perennially-reserved=true   you may create an script to mark all RDMs as perennially reserved in one go.

Confirm that the correct devices are marked as perennially reserved by running this command on the host:

#esxcli storage core device list |less

To verify about an specific lun/device, run this command:

#esxcli storage core device list -d naa.id

The configuration is permanently stored with the ESXi host and persists across restarts.

To remove the perennially reserved flag, run this command

#esxcli storage core device setconfig -d naa.id --perennially-reserved=false

To obtain LUN multipathing information from the ESXi host command line:

To get detailed information regarding the paths.

#esxcli storage core path list

or To list the detailed information of the corresponding paths for a specific device,

#esxcli storage core path list -d naa.ID

To figure out if the device is managed by VMware’s native multipath plugin, the NMP or it is managed by a third-party plugin, 

#esxcli storage nmp device list -d naa.id  
This command not only confirms that the device is managed by NMP, but will also display the Storage Array Type Plugin (SATP) for path failover and the Path Selection Policy (PSP) for load balancing.

To list LUN multipathing information,

#esxcli storage nmp device list

To check the existing path selection policy

#esxcli storage nmp satp list

To change the multipathing policy
# esxcli storage nmp device set --device naa_id --psp path_policy

(VMW_PSP_MRU or VMW_PSP_FIXED or VMW_PSP_RR)
Note: These pathing policies apply to VMware's Native Multipathing (NMP) Path Selection Plug-ins (PSP). Third-party PSPs have their own restrictions 

To generate a list of all LUN paths currently connected to the ESXi host.

#esxcli storage core path list command

For the detail path information of a specific device

#esxcli storage core path list -d naa.id

To generate a list of extents for each volume and mapping from device name to UUID,

#esxcli storage vmfs extent list command

or To generate a compact list of the LUNs currently connected to the ESXi host, including VMFS version.

#esxcli storage filesystem list

To list the possible targets for certain storage operations,

#ls -alh /vmfs/devices/disks

To rescan all HBA Adapters,
#esxcli storage core adapter rescan --all

To rescan a specific HBA.
#esxcli storage core adapter rescan --adapter <vmkernel SCSI adapter name>  Where <vmkernel SCSI adapter name> is the vmhba# to be rescanned.

To get a list of all HBA adapters,
#esxcli storage core adapter list command
Note: There may not be any output if there are no changes.

To search for new VMFS datastores, run this command,
#vmkfstools -V

To check which VAAI primitives are supported.

#esxcli storage core device vaai status get -d naa.id

The esxcli storage san namespace has some very useful commands. In the case of fiber channel you can get information about which adapters are used for FC, and display the WWNN (nodename) and WWPN (portname) information, speed and port state

#esxcli storage san fc list

To display FC event information:

# esxcli storage san fc events get

VML ID

For example: vml.02000b0000600508b4000f57fa0000400002270000485356333630
Breaking apart the VML ID for a closer understanding: The first 4 digits are VMware specific and

the next 2 digits are the LUN identifier in hexadecimal.

In the preceding example, the LUN is mapped to LUN ID 11 (hex 0b).

NAA id

NAA stands for Network Addressing Authority identifier. EUI stands for Extended Unique Identifier. The number is guaranteed to be unique to that LUN.

The NAA or EUI identifier is the preferred method of identifying LUNs and the number is generated by the storage device. Since the NAA or EUI is unique to the LUN, if the LUN is presented the same way across all ESXi hosts, the NAA or EUI identifier remains the same.

Path Identifier: vmhba<Adapter>:C<Channel>:T<Target>:L<LUN> 
This identifier is now used exclusively to identify a path to the LUN. When ESXi detects that paths associated to one LUN, each path is assigned this Path Identifier. The LUN also inherits the same name as the first path, but it is now used an a Runtime Name, and not used as readily as the above mentioned identifiers as it may be different depending on the host you are using. This identifier is generally used for operations with utilities such as vmkfstools.
Example: vmhba1:C0:T0:L0 = Adapter 1, Channel 0, Target 0, and LUN 0.

Reference: vSphere 5.x Command Line Documentation , KB# 1014953

That's it... :)