After unexpacted host reboot, Powering on a RDM attached virtual machine fails with the error: Incompatible device backing specified for device '0'

Last week one of our host unexpectedly got restarted and once the host came online we were unable to power on a VM (a passive cluster node) due to an error like,

Incompatible device backing specified for device '0'

HA didn’t restart this VM due to a VM to host-Must DRS rule.

This error occurs when LUN is not consistently mapped on hosts where primary/secondary hosts are running however here when crosschecked found everything correct (LUN Number/naa.id) on affected host.

As this was a passive node so we removed the affected drive from this VM and started this node and then started investigating the issue.

On checking the vml.id of this LUN on both hosts, found it different but the strange thing was its correct on the host in question but wrong on all other hosts in cluster. To share a LUN with different nodes, it should be consistently mapped on all hosts and should have consistent unique vml.id (VMware Legacy id) but here its different so seems the RDM disk pointer file meta data got corrupted.

You can find the vml.id of LUN as follows,

First note down/copy the identifier of LUN (naa.id) and then fire this cmd,  

#esxcli storage core device list -d naa.id

Now to fix this issue what we can do is,  remove the affected RDM disk from the both nodes and then delete the RDM pointer file from Datastore (this doesn’t affect your actual data on LUN). Now after re-scanning the hosts for Datastores, re-add the LUN as RDM drive on both nodes. Now you would be able to power on the affected node.

If due to any reason above doesn’t work then as above after removing the affected RDM drives from both nodes, follow these steps,

1. Note the NAA_ID of the LUN.
2. Detach RDM using vSphere client.
3. Un-present  the LUN from host on storage array. 
4. Rescan host storage. 
5. Remove LUN from detached list using these commands:
   
   #esxcli storage core device detached list
   #esxcli storage core device detached remove -d naa.id
6. Rescan the host storage. 
7. Re-present LUN to host. 
8. Now again rescan the hosts for datastores

If the LUN has been flagged as perennially reserved, this can prevent the removal from succeeding.

Run this command to remove the flag:

#esxcli storage core device setconfig -d naa.id --perennially-reserved=false

Now the command to remove the device should work.

# esxcli storage core device detached remove -d naa.id

Now cross check the vml.id on hosts and it should be same and after adding the RDM drive on nodes you will be able to power on the VM nodes.

Reference: VMware kb#  1016210

Update: Apr 2018

---

I didn't test it but found this work around listed in a related kb #205489

1. While adding hard disk to additional nodes of cluster, instead of selecting Existing Hard Disk under New device drop-down menu, select RDM Disk under New device drop-down menu and click Add.
2. Select the LUN naaid which was added to the first node of the cluster. The LUN number may be different on this host.
3. Verify that disk got added successfully.

That’s it… :) 

How to deal with unresponsive windows service, like vCenter svc

You might have seen this, where you tried to restart a windows service and it got stuck on stopping or in some cases starting. Recently same thing happened with me when tried to restart vCenter service, it got stuck on stopping service.

Here what we can do is, first note down the service name by going to its properties,

Here for vCenter service its ‘vpxd’

Now open windows command prompt in elevated mode and run this cmd,

C:\> sc queryex vpxd 

This will give show you the detailed info/status of intended service, note down the PID of respective service.

        SERVICE_NAME: vpxd
        TYPE               : 10  WIN32_OWN_PROCESS
        STATE              : 3  STOP_PENDING
                                (STOPPABLE, NOT_PAUSABLE, ACCEPTS_SHUTDOWN)
        WIN32_EXIT_CODE    : 0  (0x0)
        SERVICE_EXIT_CODE  : 0  (0x0)
        CHECKPOINT         : 0x0
        WAIT_HINT          : 0x493e0
        PID                : 4061
        FLAGS              :

Now run this cmd,

C:/> taskkill /f /pid xxxx

Here the PID is 4061 so,

C:/> taskkill /f /pid 4061

this will terminate the service immediately, once done then you can start the service either from GUI (services console) or from cmd itself by running this command, 

C:/> sc start vpxd

That’s it... :)

How to find esxi host install date

Last year I asked the same question on VMTN but could find a proper answer or way to find the original install date of an ESXi host until now. Earlier I was able to find the last upgrade/patching date by using the following commands but not the esxi install date,

#esxcli software vib list | grep -i base

or #esxcli software profile get

or using PowerCLI command,

Get-VMHostPatch *| Select ID, InstallDate, VMHostID

Note: Above commands can serve the purpose only if one didn’t install any patches or upgrade, otherwise you will get the last patch/upgrade installation date.

Today I came across a newly posted KB article #2144905 and here I found exactly what I was looking for so thought of making a note of it for future reference.

To determine the installation date of the ESXi host what you need to do is, first connect to the intended host over ssh using Putty and run the following command,

#echo -n "ESXi install date: " ; date -d @$(printf "%d" 0x$(esxcli system uuid get | cut -d \- -f1 ))

As you can see here we got the ESXi host installation date.

As one can’t remember this command all the time ;) so alternatively you can do this... get the system uid by using this command, esxcli system uuid get, here the first part of the system uid is the UNIX time in hexadecimal at the time the UUID was created (or esxi was installed), now convert this hexadecimal value to decimal (using any online converter) and now convert the decimal vale to a readable date format using any UNIX online timestams converter, .

That’s it... :)

VMware announced retirement plan for vSphere Client aka Desktop Client

Today VMware announced the retirement plan of vSphere Client aka Desktop Client, from the future releases of VMware vSphere there will be only HTML5 based Web client and no desktop client.

Current versions of vSphere (5.5 or 6.0) will not be affected and follow the standard support period.

For detailed information about this announcement please refer to following blog post on VMware blogs,
Goodbye vSphere Client for Windows (C#) – Hello HTML5

Currently HTML5 based Web client is available as a fling and its OVA can be downloaded from fligs by VMware Site .

Lets hope by the time of general release of vSphere HTML5 Web Client, it would be as good and snappy as vSphere desktop client.

That's it... :)

How to view more/old vCenter "Task & Events"

As most of us would be aware, by default one would see only 100 entries in vCenter Task & Events and sometimes while troubleshooting an issue that is not enough and you want to see previous/more events to find the time of issue.

There is another situation where you are looking into an issue in vCenter and the time the issue happened has just dropped off of the bottom of the Tasks or Events lists?

Well we can change the number of task & events to displayed using vSphare client, to do so,

First connect to vCenter or a host using vSphere Client => Client Settings

On this windows, Click on Lists Tab, here is the Setting we are looking at is the Page Size,

Now increase the value of Page Size from default value 100 to any value for example 1000 and you are done and can see more entries in Task & Events.

The actual Task & Event data to be displayed is depend on the vCenter Database retention policy and it is set to 180 days by default and can be chages to desiresd value in days. In case you are changing it ,then bear in mind the fact it would increase or decrease the size of vCenter database accordingly. 

To change the vCenter database retention policy, 

In vSphere client, Go to vCenter Home => Select vCenter Settings => Now click on Database retention policy

From here you can increase or decrease the same.

Related useful kb# 2110031

That’s it… J

New VMware Fling: vSphere HTML5 Web Client

Last day while looking at VMwareLabs Flings site, came across this vSphere HTML5 Web Client fling. Its good to see some progress towards HTML5 based web client, first VMware engineers came up with HTML5 based embedded host client and now this.
As per the given info, this web client is written using HTML5 and Javascript and available for kind of beta testing however this Fling is not fully complete yet.

As per the team behind this fling, we wanted to get it in front of our customers as soon as possible, and so we are only offering the following features for the time being (we feel that these are the most commonly used actions/views):

• VM Power Operations (common cases)
• VM Edit Settings (simple CPU, Memory, Disk changes)
• VM Console
• VM and Host Summary pages
• VM Migration (only to a Host)
• Clone to Template/VM
• Create VM on a Host (limited)
• Additional monitoring views (Performance charts, Tasks, Events)
• Global Views (Recent tasks, Alarms–view only)

This Fling has been designed to work with your existing vSphere 6.0 environments. The new client is deployed as a new VM from the downloadable OVA.  Currently the installation instructions are command line-based, but we are working on a GUI installation and plan to release it as an update to this Fling once it is ready.
We intend to regularly update this Fling to both provide new features and address reported issues, based on feedback.

Are you ready? Click here to get started with the vSphere HTML5 Web Client.  

Make sure to check out the list of known issues and notes before jumping in.


That's it... :)

ESXi 5.x host not accessible/showing as inaccessible after reboot

You might have seen this issue where you rebooted an host and even after waiting for a long time it didn't come up and showing as inaccessible in vCenter inventory. Tried  to reconnect the host but end up with this error, 
"Cannot contact the specified host (ESXi1.mylab.com). The host may not be available on the network, a network configurati-on problem may exist, or the manageme-nt services on this host may not be responding."

Then I tried to ping this host but no luck, however when connected to host using physical server remote management tool, found the host up.

This is something that I had seen earlier, in this case what you need to check is, login to DCUI from server remote management console (hp iLO or Dell iDRAC or IBM IMM or whatever) and then check IP configuration of the host as well as test the management network.

Most of the time when you test the management network, you would see something like, host is not reaching to gateway/DNS or sometimes strangely DNS is not reachable but name resolution is happening.

In this case most of the time, fix of this issue as simple as restarting the management network.

  
Sometimes you may need to restart the management network more than once.

As soon as management network restart, host would be accessible (start pinging) again however sometimes you may also need to restart host management agents to make it available/accessible from vCenter console.

That's it... :) 

VMware Host Client uility: ESXi Embedded Host Client

The VMware Host Client is an HTML5-based client that is used to connect to and manage single ESXi host directly from web browser. It can be used to perform administrative tasks to manage host resources such as virtual machines, networking, and storage. The VMware Host Client can also be helpful for troubleshooting individual virtual machines or hosts if vCenter Server and the vSphere Client are unavailable.

The ESXi Host Client is part of ESXi 6.0 Update 2 now: Earlier it was available as a ESXi Embedded Host Client Fling only, so we have to install it before connecting to host via browser (supported for ESXi5.x and later).
The utility is bundled as a vib and can be installed on ESXi host via SSH without the need to reboot the host.
To use it on earlier versions of ESXi (prior to ESXi 6.0 Update 2), download and upload the Host Client VIB to any accessible datastore and then connect to host using putty and run following command,
#esxcli software vib install -d /vmfs/volumes/datastore1/esxui-signed-3623722.vib  

In case you downloaded the offline bundle, then

#esxcli software vib install -d /vmfs/volumes/datastore1/esxui-offline-bundle-5.x-3623722.zip

Once the VIB is installed, you simply just point your web browser to the following URL to access the host,

https://ESXi_HostName_or_IP/ui/ , For ESXi 5.5U2 and prior, and ESXi 6.0 hosts upgraded from any 5.5U2 or prior version, you will get a 503 error returned after visiting https://ESXi_HostName_or_IP/ui/. its a known issue and to resolve it, please remove the line starting with /ui from /etc/vmware/rhttpproxy/endpoints.conf and restart rhttpproxy with /etc/init.d/rhttpproxy restart
For ESXi 5.5U2 and prior hosts, you must ensure to append a trailing / (forward slash) to the URL after /ui. The URL must be https://ESXi_HostName_or_IP/ui/

In case if you want to remove the ESXi embedded host client from your ESXi host, then

#esxcli software vib remove -n esx-ui

In vSphere 6.0 Update 2, VMware continues to support the vSphere Client, but you now have the option of using the VMware Host Client instead of the vSphere Client to perform similar host-based operations.

VMware Host Client Features

The VMware Host Client 1.0.0 functions include, but are not limited to the following operations:

• Support for the latest hardware version.
• Basic virtualization operations, such as deploying, configuring, and editing virtual machines of various complexity, including console access to the virtual machines.
• Creating and managing network and datastore resources.
• Displaying current host and resource settings, including graphs of performance and utilization, and logs of host components to help troubleshooting.
• Advanced tuning of host level options to improve performance.

Full Release Notes is available here: VMware Host Client Release Notes

That's it... :)