vCenter 6 : Inventory search failed with the error, login to query service failed

We recently upgraded to vCenter 6.0 and today when I tried to search for virtual machines with the vSphere Client 6, the search failed with this error,

*I didn't see this behavior when checked via Web Client.

Initially I thought, its due to some DNS issue however when checked, was able to ping the vCenter server with its FQDN.

On further checking, came across VMware KB# 2143566, which is about the exact same issue.

As per this KB,

This is an expected behavior.

Searching for Inventory objects when logged in to the vSphere Client using the Use Windows session credentials option is not supported.

To work around this issue, log in to the vSphere Client by manually entering the login credentials.

Note: Using the same vSphere 6 client, I didn't see any issue when connected to vCenter 5.5 so this is something expected with vSphere 6.0.

That’s it… :)

VM crashed during backup and now getting error when trying to power on : sparse disk

Today while taking backup of a Linux VM, it got crashed during backup however backup completed successfully. When tried to power on this VM, end up with this error,

An error was received from the ESX host while powering on VM Test-VM01

Cannot open the disk '/vmfs/volumes/5e4eb348-00341c9a-a87d-011d296158e2/Test-VM01/Test-VM01-000001.vmdk' or one of the snapshot disks it depends on.

The system cannot find the file specified

VMware ESX cannot find the virtual disk "/vmfs/volumes/5e4eb348-00341c9a-a87d-011d296158e2/Test-VM01/Test-VM01-000001.vmdk". Verify the path is valid and try again.

Here I was able to quickly get the VM back in working condition by just pointing the disk back to original disk file however now I noticed this VM is using a Sparse disk which is spirited into 2Gbsize files.

This kind are disks are disallowed on all ESXi 5.x and later hosts and ideally we wouldn't able to power on VMs with such disk (VMware kb#1022242) however here I was/am able to power on this VM, noticed any issue only after backup.

Here one thing is worth to point out, this VM was not created on Esxi 5.x host but was imported from Virtual Box to VMware environment.

To fix this issue, power off the VM and clone all the sparse disks to a compatible format with vmkfstools by running this command,

# vmkfstools -i /vmfs/vloumes/.../source/VM_Name.vmdk -d disk_format(thin/zeroedthick/eagerzeroedthick) /vmfs/vloumes/.../destination/VM_Name.vmdk

Once completed then from VM edit settings, remove the old sparse disk and add this thin/thick vmdk file and now you can power on the VM.

As above process required downtime so alternatively I tried storage vMotion and it worked for me...so here the preferred fix is.. just Storage vMotion the affected VM to another Datastore, this will change the disk format on the fly and we are done.

Then just to check if we can restore the VM backup which was taken using Dell vRanger backup, when we tried to restore the backup it failed with the error: Error: API Call failed with message: The device or operation specified at index '9' is not supported for the existing virtual machine platform.

When checked about the error found a kb article on Dell support site, kb#104897, 19559960, according to this kb, we may take backup of a such kind of VM running on ESXi host but wouldn’t be able restore the same, and in order to fix this behavior we need to change the Sparse disk to thin/thick VMDK file. The process of converting the sparse disk is same a described above.

That's it... :)

Intermittent network connectivity issue with a Physical server having physically disconnected NICs in NIC Team

Last week I came across this issue where a physical server was losing network connectivity every now and then.

When checked, found this Windows 2012 server having four network cards but out of four only two were connected to network and for other two NICs, network cable was unplugged.

When checked the network teaming properties,

Server manger => Local Server =>from right side console, NIC Teaming and click on enabled => now under Team, select your NIC Team and go to its properties

Here as you can see,  disconnected NIC cards were also part of “NIC Team” so we suspect due to this the dynamic load balancing is having issues and so the server having Intermittent network connectivity problem.

We removed the disconnected NIC cards from NIC Teaming (just uncheck and click ok) and from then to till now didn't hear back any network connectivity issue for this server. 

Please share your thoughts in comment area if thinks otherwise and have any other solution. 

That's it... :)

Unable to add host to vCenter, getting error like: The evaluation Mode license assigned to Host has expired

Last day I received a question from one of my friend, in his lab environment he was trying to add an ESXi host to vCenter inventory but was unable to do so (both host and vCenter are at 6.0 update 2) and was getting an error like,

"The evaluation Mode license assigned to Host hostname has expired. Recommend updating the license"

As he was just configuring the lab so of course the trial period was not expired so what’s the issue here.

The other thing which can cause this kind of issue is system date and time, there is a VMware kb# 2011655, which covers this issue.

This issue occurs if the date and time in the BIOS is incorrect.

When you add an ESXi/ESX host with an incorrect system time to vCenter Server, the license is converted to an evaluation license that immediately expires.

To resolve this issue, one need to correct the date and time in the host BIOS, once done then you will be able to add the host to the vCenter Server inventory.

That’s it… :)

HP ProLiant G8/G9 Servers, firmware bug: the BIOS has corrupted hw-PMU resources

Last week while installing ESXi 6.x on HP ProLiant G9 server, initially I selected intelligent provisioning and during POST I came across this error message, "[Firmware Bug]: The BIOS Has Corrupted Hw-PMU Resources.

Note: Even with normal install, you may see this error during POST.

Initially I thought there is some issue with firmware so tried to update the firmware using latest HP SPP but found everything already up-to-date. When checked with HP support about this issue, they said it’s a known issue and can be safely ignored.

As per them, “BIOS has corrupted Hw-PMU” error is of no concern and is only the Linux kernel noticing some performance counter registers in the processor had been initialized.

And as listed on this blog,

It is just an informational message. The system is reserving performance counters for system management and the kernel wants to own all the performance counters regardless. You can disable the ProLiant management features if you really want to stop the message. Other than printing the message during boot, there isn’t any negative impact on the system or performance.

As a way around onecan avoid this error msg by disabling the option of Processor Power and Utilization Monitoring in BIOS (enabled by default) but please keep in mind, disabling this would also disable some iLO/CPU monitoring/reporting features (for detail check here, page 10).

To disable Processor Power and Utilization Monitoring in BIOS do the following,

During POST, press F9 to enter BIOS => press CTRL+A (Service Option is hidden by default) => select “Service Options” => Processor Power and Utilization Monitoring and disable it
Now press F10 to save and exit and then reboot the server.

Related Support Information: HP ProLiant DL360/DL380 Gen9 Server - Install with Red Hat Enterprise Server Version 6.6 Gives Error During Post : The BIOS has corrupted hw-PMU Resources (MSR 38d is 330) 

That’s it… :)

FreeNAS Web interface connection Error: Welcome to nginx!

Today while re-building my vSphere6.x HomeLab, after installing the FreeNAS, when tried to connect to FreeNAS Web interface found it just showing this:

In this case you can surpass this by running this cmd on FreeNAS console,

# service ix-nginx start 
#nginx

If same issue still persist, try this,

# rm /usr/local/etc/nginx/nginx.conf*
# service ix-nginx start
# nginx

Reference: FreeNAS Forum

That's it... :)

Unmounting a LUN or detaching a datastore from ESXi 5.x or 6.0

You might have seen where you unmounted/detached a LUN from ESXi host and after some time few hosts in your env are showing as Inaccessible/Not-Responding. When you would further check the VMkernel logs on affected host, you would find APD/PDL related log entries.

This is something that could be related to not following the proper procedure during LUN detach. If you wouldn’t follow the proper process during LUN/Datastore unmount/detach then it could lead the host to APD/PDL state.

In this post I will summarizing the best practice of unmounting a LUN from ESXi 5.x or 6.

Before doing anything, ensure that:

• Host should not have any registered virtual machines/template residing on this datastore and all CD/DVD images located on the VMFS datastore must also be unregistered from any virtual machines.
• The datastore is not used for vSphere HA heartbeat.
• The datastore is not part of a datastore cluster (managed by Storage DRS).
• The datastore is not configured as a diagnostic coredump partition.
• Storage I/O Control is disabled for the datastore.
• If the LUN is being used as an RDM, remove the RDM from the virtual machine. Click Edit Settings, highlight the RDM hard disk, and click Remove. Select Delete from disk and click OK.
  
  Note: This destroys the mapping file but not the LUN content.
• Check if the LUN/datastore is used as the persistent scratch location for the host.
  
  

Note: When using the vSphere Web Client with vSphere 5.1, 5.5, and 6.0, only following checks are required during datastore unmount,

• Host should not have any virtual machines residing on this datastore
• Host should not use the datastore for HA heartbeats

Obtaining the NAA ID of the LUN to be removed

From the vSphere Client, this information is visible in the Properties window of the datastore.

From the ESXi host, run this command:

# esxcli storage vmfs extent list

You see output similar to:

Volume Name  VMFS  UUID  Extent Number  Device  Name Partition

----------- ----------------------------------- ------------- ------------------------------------ ---------

datastore1  4de4cb24-4cff750f-85f5-0019b9f1ecf6 0  naa.6001c230d8abfe000ff76c198ddbc13e 3

Storage2  4c5fbff6-f4069088-af4f-0019b9f1ecf4 0  naa.6001c230d8abfe000ff76c2e7384fc9a 1

Storage4  4c5fc023-ea0d4203-8517-0019b9f1ecf4 0  naa.6001c230d8abfe000ff76c51486715db 1

LUN01  4e414917-a8d75514-6bae-0019b9f1ecf4 0  naa.60a98000572d54724a34655733506751 1

Make a note of the NAA ID of the datastore to use this information later in this procedure.

Note: Alternatively, you can run the esxcli storage filesystem list command, which lists all file systems recognized by the ESXi host. To find the unique identifier of the LUN housing the datastore to be removed, run this command:

# esxcfg-scsidevs –m 

This command generates a list of VMFS datastore volumes and their related unique identifiers. Make a note of the unique identifier (NAA_ID) for the datastore you want to unmount as this will be used later on.

Unmounting and de a LUN using the vSphere Client

To detach a storage device using the vSphere Client, first unmount the datastore and then detach the LUN, process is as follows,

1.       If the LUN is an RDM, skip to step 2. Otherwise, in the Configuration tab of the ESXi host, click Storage. Right-click the datastore being removed and click Unmount.

A Confirm Datastore Unmount window appears. When the prerequisite criteria have been passed, click OK.

Note: To unmount a datastore from multiple hosts in the vSphere Client, click Hosts and Clusters > Datastores and Datastore Clusters view (Ctrl+Shift+D). Perform the unmount task and select the appropriate hosts that should no longer access the datastore to be unmounted.

2.     Click the Devices view (under Configuration > Storage):

3.       Right-click the NAA ID of the LUN (as noted above) and click Detach. A Confirm Device Unmount window is displayed. When the prerequisite criteria are passed, click OK. Under the Operational State of the Device, the LUN is listed as Unmounted.

Note: The Detach function must be performed on a per-host basis and does not propagate to other hosts in vCenter Server. If a LUN is presented to an initiator group or storage group on the SAN, the Detach function must be performed on every host in that initiator group before unmapping the LUN from the group on the SAN. Failing to follow this step results in an all-paths-down (APD) state for those hosts in the storage group on which Detach was not performed for the LUN being unmapped.

4.       Confirm if the LUN is successfully detached. The LUN can then be safely unpresented from the SAN.

5.       Perform a rescan on all ESXi hosts which had visibility to the LUN. The device is automatically removed from the Storage Adapters.

When the device is detached, it stays in an unmounted state even if the device is re-presented (that is, the detached state is persistent). To bring the device back online, the device must be attached.

If you want the device to permanently decommission from an ESXi host, manually remove the NAA entries from the host configuration:

·         To list the permanently detached devices, run this command:

# esxcli storage core device detached list

You see output similar to:

Device UID State

------------------------------------ -----

naa.50060160c46036df50060160c46036df off

naa.6006016094602800c8e3e1c5d3c8e011 off 

·         To permanently remove the device configuration information from the system, run this command:

# esxcli storage core device detached remove -d NAA_ID

For example:

# esxcli storage core device detached remove -d naa.50060160c46036df50060160c46036df

This is it.

Unmounting a LUN using the command line

To unmount a LUN from an ESXi 5.x/6.0 host using the command line:

• As earlier, obtain the NAA ID of the LUN to be removed
• Now unmount the datastore by running this command:
  
  # esxcli storage filesystem unmount [-u UUID | -l label | -p path ]
  
  For example, use one of these commands to unmount the LUN01 datastore:
  
  # esxcli storage filesystem unmount -l LUN01
  # esxcli storage filesystem unmount -u 4e414917-a8d75514-6bae-0019b9f1ecf4
  # esxcli storage filesystem unmount -p /vmfs/volumes/4e414917-a8d75514-6bae-0019b9f1ecf4
  
  Note: If the VMFS filesystem you are attempting to unmount has active I/O or has not fulfilled the prerequisites to unmount the VMFS datastore, you see an error in the VMkernel logs similar to:
  
  WARNING: VC: 637: unmounting opened volume ('4e414917-a8d75514-6bae-0019b9f1ecf4' 'LUN01') is not allowed.
  VC: 802: Unmount VMFS volume f530 28 2 4e414917a8d7551419006bae f4ecf19b 4 1 0 0 0 0 0 : Busy
• To verify that the datastore is unmounted, run this command:
  
  # esxcli storage filesystem list
  
  You see output similar to:
  
  Mount Point  Volume Name  UUID  Mounted  Type  Size  Free
  ------------------------------------------------- ----------- ----------------------------------- ------- ------ ----------- -----------
  /vmfs/volumes/4de4cb24-4cff750f-85f5-0019b9f1ecf6  datastore1  4de4cb24-4cff750f-85f5-0019b9f1ecf6  true  VMFS-5  140660178944  94577360896
  /vmfs/volumes/4c5fbff6-f4069088-af4f-0019b9f1ecf4  Storage2  4c5fbff6-f4069088-af4f-0019b9f1ecf4  true  VMFS-3  146028888064  7968129024
  /vmfs/volumes/4c5fc023-ea0d4203-8517-0019b9f1ecf4  Storage4  4c5fc023-ea0d4203-8517-0019b9f1ecf4  true  VMFS-3  146028888064  121057050624
  LUN01  4e414917-a8d75514-6bae-0019b9f1ecf4  false VMFS-unknown  version 0 0
  
  The Mounted field is set to false, the Type field is set to VMFS-unknown version, and that no Mount Point exists.
  
  Note: The unmounted state of the VMFS datastore persists across reboots. This is the default behavior. If you need to unmount a datastore temporarily, you can do so by appending the --no-persist flag to the unmount command.
• To detach the device/LUN, run this command:
  
  # esxcli storage core device set --state=off -d NAA_ID
• To verify that the device is offline, run this command:
  
  # esxcli storage core device list -d NAA_ID
  
  You see output, which shows that the Status of the disk is off, similar to:
  
  naa.60a98000572d54724a34655733506751
  Display Name: NETAPP Fibre Channel Disk (naa.60a98000572d54724a34655733506751)
  Has Settable Display Name: true
  Size: 1048593
  Device Type: Direct-Access
  Multipath Plugin: NMP
  Devfs Path: /vmfs/devices/disks/naa.60a98000572d54724a34655733506751
  Vendor: NETAPP
  Model: LUN
  Revision: 7330
  SCSI Level: 4
  Is Pseudo: false
  Status: off
  Is RDM Capable: true
  Is Local: false
  Is Removable: false
  Is SSD: false
  Is Offline: false
  Is Perennially Reserved: false
  Thin Provisioning Status: yes
  Attached Filters:
  VAAI Status: unknown
  Other UIDs: vml.020000000060a98000572d54724a346557335067514c554e202020

This device is now successfully detached from the host. It remains visible the UI at this point.

If the device is to be permanently decommissioned, it is now possible to unpresent the LUN from the SAN.

• To rescan all devices on the ESXi host, run this command:
  
  # esxcli storage core adapter rescan [ -A vmhba# | --all ]
  
  The devices are automatically removed from the Storage Adapters.
  
  Notes:
• A rescan must be run on all hosts that had visibility to the removed LUN.
• When the device is detached, it stays in an unmounted state even if the device is re-presented (that is, the detached state is persistent). To bring the device back online, the device must be attached. To do this via the command line, run this command:
  
  # esxcli storage core device set --state=on -d NAA_ID
• If the device is to be permanently decommissioned from an ESXi host, (that is, the LUN has been or will be destroyed), remove the NAA entries from the host configuration by running these commands:
• To list the permanently detached devices:
  
  # esxcli storage core device detached list
  
  You see output similar to:
  
  Device UID State
  ---------------------------- -----
  naa.50060160c46036df50060160c46036df off
  naa.6006016094602800c8e3e1c5d3c8e011 off
• To permanently remove the device configuration information from the system:
  
  # esxcli storage core device detached remove -d NAA_ID
  
  For example:
  
  # esxcli storage core device detached remove -d naa.50060160c46036df50060160c46036df
• The reference to the device configuration is permanently removed from the ESXi host's configuration.
  
  Note: If the device is detached but still presented (highlighted step was skipped), the preceding command fails to permanently remove the device from the system, and the device is automatically re-attached. You must complete highlighted step for the device to be permanently removed.

Reference: VMware KB# 2004605, 2004684

That's it... :)

After unexpacted host reboot, Powering on a RDM attached virtual machine fails with the error: Incompatible device backing specified for device '0'

Last week one of our host unexpectedly got restarted and once the host came online we were unable to power on a VM (a passive cluster node) due to an error like,

Incompatible device backing specified for device '0'

HA didn’t restart this VM due to a VM to host-Must DRS rule.

This error occurs when LUN is not consistently mapped on hosts where primary/secondary hosts are running however here when crosschecked found everything correct (LUN Number/naa.id) on affected host.

As this was a passive node so we removed the affected drive from this VM and started this node and then started investigating the issue.

On checking the vml.id of this LUN on both hosts, found it different but the strange thing was its correct on the host in question but wrong on all other hosts in cluster. To share a LUN with different nodes, it should be consistently mapped on all hosts and should have consistent unique vml.id (VMware Legacy id) but here its different so seems the RDM disk pointer file meta data got corrupted.

You can find the vml.id of LUN as follows,

First note down/copy the identifier of LUN (naa.id) and then fire this cmd,  

#esxcli storage core device list -d naa.id

Now to fix this issue what we can do is,  remove the affected RDM disk from the both nodes and then delete the RDM pointer file from Datastore (this doesn’t affect your actual data on LUN). Now after re-scanning the hosts for Datastores, re-add the LUN as RDM drive on both nodes. Now you would be able to power on the affected node.

If due to any reason above doesn’t work then as above after removing the affected RDM drives from both nodes, follow these steps,

1. Note the NAA_ID of the LUN.
2. Detach RDM using vSphere client.
3. Un-present  the LUN from host on storage array. 
4. Rescan host storage. 
5. Remove LUN from detached list using these commands:
   
   #esxcli storage core device detached list
   #esxcli storage core device detached remove -d naa.id
6. Rescan the host storage. 
7. Re-present LUN to host. 
8. Now again rescan the hosts for datastores

If the LUN has been flagged as perennially reserved, this can prevent the removal from succeeding.

Run this command to remove the flag:

#esxcli storage core device setconfig -d naa.id --perennially-reserved=false

Now the command to remove the device should work.

# esxcli storage core device detached remove -d naa.id

Now cross check the vml.id on hosts and it should be same and after adding the RDM drive on nodes you will be able to power on the VM nodes.

Reference: VMware kb#  1016210

Update: Apr 2018

---

I didn't test it but found this work around listed in a related kb #205489

1. While adding hard disk to additional nodes of cluster, instead of selecting Existing Hard Disk under New device drop-down menu, select RDM Disk under New device drop-down menu and click Add.
2. Select the LUN naaid which was added to the first node of the cluster. The LUN number may be different on this host.
3. Verify that disk got added successfully.

That’s it… :)