How to find and install VMware PowerCLI Module

As most of us would be already aware that, VMware PowerCLI is now available as a PS module and can be installed directly from Microsoft Powershell Gallery, in this post I'll summaries the process of online and offline installation of PowerCLI.

PS commands to be used:

Get-Module : To list the available/already installed modules on you local machine
Find-Module : To find the available modules in an Online PS Gallery
Install-Module : To download and install the specified module from an online gallery
Save-Module : To download and sad save a PS module for offline use

Online installation of PowerCLI : To install the PowerCLI from Microsoft PS gallery, follow these steps,

• If you are not sure about the name of the actual VMware PowerCLI module, use below cmd to find the same,

Find-Module -name *VMware*

• Once you have identified the module name, use Install-Module CMD to install it.

          Here we will install VMware "VMware.PowerCLI"

Install-Module -Name VMware.PowerCLI

Press Y [yes] or A[yes to all], when prompted.

Depending upon your internet connection speed, it would take some time in downloading and installing the requested PS module, once done then close the current Powershell sessions and restart it, now newly installed PowerCLI module will load and you can use it.

In case if you want to limit the scope of this module to current-user only then use following command instead, 

Install-Module -Name VMware.PowerCLI –Scope CurrentUser

Offline installation of PowerCLI: To download and install VMware PowerCLI Module follow these steps,

• On a a system which is connected to internet, you can save the VMware PowerCLI module to your desired location using Save-Module cmd.

• Now, we can copy this module (downloaded folders) to a system which is not connected to internet, and place them in a location where PowerShell can find them, modules folders can be confirmed by typing $ENV:PSModulePath at the PowerShell prompt.

Local User: $home\Documents\WindowsPowerShell\Modules 

All Users: $pshome\Modules

Once you copied these folder to the the modules folders and restarted the PowerShell window, the modules will automatically load this time and you can use them now.

Reference: PowerCLI Installation Walkthrough

That's it... :)

How to reset iLO login password from within ESXi Shell

This is something could be useful if you are unable to recall the HP iLO login password.

Follow these steps to reset iLO password from esxi host: 

Create a file named reset_ilo_pwd.xml in tmp folder under root with the below info and add the new password in the password section:

<ribcl VERSION=”2.0″>
<login USER_LOGIN=”Administrator” PASSWORD=”password“>
<user_INFO MODE=”write”>
<mod_USER USER_LOGIN=”Administrator”>
<password value=”YOUR-NEW-PASSWORD”/>
</mod_USER>
</user_INFO>
</login>
</ribcl>

Alternatively you can create this file by copying these lines to notepad and then saving it as reset_ilo_pwd.xml file, now upload it to /tmp directory of your ESXi host using any ftp client like Winscp.

Now use the following command to commit the administrator password information from the file (reset_ilo_pwd.xml) to iLO using the below command

 ./hponcfg -f /tmp/reset_ilo_pwd.xml

This is it, now you should be able to login to iLO of your ESXi host using the password specified in the above xml file.

Additional Info: HPE offers support for the iLO features available on ProLiant servers with the HP Online iLO Configuration aka HPONCFG utility. HPONCFG can be used to set up and reconfigure the iLO (Integrated Lights-Out) management controller of a server.

HPONCFG is an online configuration tool used to set up and reconfigure iLO without requiring a reboot of the server operating system. The utility runs in a command-line mode and must be executed from an operating system command line on the local server. HPONCFG enables you to initially configure features exposed through the RBSU or iLO.

You can also use HPONCFG to reset the ILO from ESXi host and many other things.

Browse to /opt/hp/tools and execute the below command to reset the HP ILO configuration.

./hponcfg -r 

Before using HPONCFG, the iLO Management Interface Driver must be loaded on the server. HPONCFG displays a warning if the driver is not installed.

In case if its not already present on the server, you may download and install from following links: 

* HPE ESXi Utilities Offline Bundle for VMware vSphere 6.0 *

https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_296f7134edb64f83822c7de198#tab-history

* HPE ESXi Utilities Offline Bundle for VMware vSphere 6.5 *

https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_6be6cedecfee4ffe9f6c808711#tab-history 

That's it... :)

How to reset HP iLO over ssh or remotely

Lately I faced this issue, where I was unable to login to iLO of one of my HP ProLiant server instead there was a ssh connection related error on login page.

I had seen this error during the iLO firmware upgrade in past but not any other time.

As I did upgrade this ProLiant server’s Firmware/Drivers last day using the latest HP Service Pack forProLiant (aka HP SPP) so, it had the latest iLO firmware ver 2.55 installed and I was sure everything was working as expected post SPP upgrade.

We tried few things to resolve the issue and eventually resting the iLO fixed the issue, as iLO web login was not available so we had to reset it via command line by connecting to iLO over ssh.

You would be able to access iLO over ssh only if you didn’t disable it during the initial iLO configuration (By default iLO ssh access is enabled).

The steps to reset the HP iLO over ssh are follows:-

1. Open Putty (or any other ssh client) and connect to the iLO controller using its IP address or FQDN
2. Login with iLO administrator credentails (or an account with equivalent rights)
3. Once connected to iLO, issue the following command:

   cd /map1

Press Enter

4. Then type

   Reset

Press Enter

Command and its output would be something like this on CLI:

hpiLO-> cd /map1

status=0
status_tag=COMMAND COMPLETED

hpiLO-> reset

status=0
status_tag=COMMAND COMPLETED

Resetting iLO.

CLI session stopped

The iLO controller will begin to reset itself. Resetting the controller fixes a lot of hung-up and freezes issues on the interface.

Note: In case of any issue, HPE always recommends upgrading the firmware on the iLO controller as the first thing and as it doesn’t require reboots or downtime so can be done anytime.

Additional resources:   

HPE iLO 4 Scripting and Command Line Guide - HPE Support Center

http://www.softpanorama.org/Hardware/HP/ILO/command_line_interface.shtml 

https://community.hpe.com/t5/ProLiant-Servers-ML-DL-SL/DL360p-gen8-ilo-4-not-available-responding/m-p/6987976#M159786

That's it... :)

Extended LUN size but unable to increase the datastore size from vCenter

You might have seen this issue where in vCenter managed environment, you want to expand the datastore size and Storage team has confirmed that they have increased the respective LUN size and the same is reflecting in datastore properties under Device capacity however when you click on increase under Volume Properties, there is no extent available.

I had seen such issue in past but when adding additional extents, not when increasing datastore size.

vCenter Server calls a specific function to get all the available extents for that datastore. After getting extents, vCenter Server displays the extents as available if they meet these criteria filters:

• LUNS are not used as datastores on that host or on any other host (with exceptions to force mounted volumes).
• LUNS are not used as Raw Device Maps(RDMs) on that host or any other host.

vCenter Server applies these filters to stop any possible data corruption on already used LUNs by other host.

VMware recommends you to not to turn off the filters. Before making any changes to the LUN filters, contact the VMware Support team. 

When you connect directly to the ESX host, these filters do not exist and hence you can see the extents.

  

To work around this issue, you need to connect to the ESXi directly using 'root' credentials and then increase the volume size from there, once done then re-scan all ESxi hosts for storage. 

Once you do that the expended datastore would be visible to all other ESXi hosts as well. 

Related VMware KB article :- 1011754

That's it... :)

Conflicting VIBs error during ESXi upgrade using ISO image

You might have came across this error where, trying to upgrade the ESXi host using OEM provided Custom ESXi image or may be with VMware provided ESXi image and the upgrade got aborted with the CONFLICTING_VIBs ERROR.

In order to successfully complete the ESXi upgrade, one need to to do the following:

• Connect to host over ssh using putty and run the following command find the exact name of vib

       # esxcli software vib list | grep “displayed conflicting vib”

•  Once you identified the exact vib name, then just remove it by using following command

       #esxcli software vib remove -n “actual name of conflicting vib”

Note: Please do an impact analysis before you proceed with vib removal.

Now you boot up the host using ESXi ISO image and complete the upgrade (this time you shouldn’t see earlier previous conflicting vib error).

That's it... :)

Intel / AMD processor vulnerability: Meltdown-Spectre and VMware Esxi

Most of us would be aware about this by now...if not, there were serious security flaws named Meltdown and Spectre discovered in processors designed by Intel, AMD and ARM, these flaws could let attackers steal your sensitive data.

These flaws were discovered by security researchers at Google’s Project Zero in conjunction with academic and industry researchers from several countries. Combined they affect virtually every modern computer, including smartphones, tablets and PCs from all vendors and running almost any operating system like Windows, macOS and Linux etc.

The two ‘bugs’ stem from design flaws of microprocessors that have the potential to allow applications, malware, and JavaScript running in web browsers, to obtain information from the operating system kernel’s private memory areas.

So here you may think how would it affect the Vmware Esxi platform and the VMs running on it.

VMware has issued a Security Advisory (VMSA-2018-0002) for the same and according it, CPU data cache timing can be abused to efficiently leak information out of miss-speculated CPU execution, leading to (at worst) arbitrary virtual memory read vulnerabilities across local security boundaries in various contexts. (Speculative execution is an automatic and inherent CPU performance optimization used in all modern processors.) ESXi, Workstation and Fusion are vulnerable to Bounds Check Bypass and Branch Target Injection issues resulting from this vulnerability.

Result of exploitation may allow for information disclosure from one Virtual Machine to another Virtual Machine that is running on the same host.

To remediate the observed vulnerability (known variants of the Bounds Check Bypass and Branch Target Injection issues) in each in different versions of Esxi releases, one need to install a corresponding patch from the list.

VMware Patches for different  ESXi Versions:

• ESXi 6.5 – ESXi650-201712101-SG,    There are new patches available,
• ESXi 6.0 – ESXi600-201711101-SG,    Refer to following Advisory VMSA-2018-0004
• ESXi 5.5 – ESXi550-201709101-SG *

* This patch has remediation against CVE-2017-5715 but not against CVE-2017-5753.

Downloads:  https://my.vmware.com/group/vmware/patch, Search with the patch name.

Whilst this will secure the risk of data leakage between virtual machines it will not mitigate against the risk of data leakage within individual virtual machines. To protect against this threat operating system specific security updates must be installed.
Microsoft has already released a patch on Jan 3rd, 2018 to fix this issue on systems running on Windows OSs.

Also Apply the applicable firmware update provided by your server/device manufacturer, Useful Link.

Note: It has been speculated that patching the flaws would cause performance hit. At this time, it’s still unclear what would be the degree of performance hit, currently the details available varying with the source of information.

Related Read: https://www.pcworld.com/article/3245606/security/intel-x86-cpu-kernel-bug-faq-how-it-affects-pc-mac.html
https://www.theverge.com/2018/1/4/16848976/how-to-protect-windows-pc-meltdown-security-flaw

That’s it…. 😊

How to capture memory dump of a VM from snapshot or suspended state file

This is something could be requested by an application vendor for debugging purpose to investigate an application related issue. If you get any such request for VM memory dump then you might wonder how to capture a memory dump from a VMware virtual machine without stopping its execution.

If this is a production VM then you might not want to force a crash or change Windows dump parameters and reboot the machine. So, how can we capture a memory dump of a VM without interrupting it.

There is a VMware fling called vmss2core, using which we can convert the checkpoint state files into formats that third party debugger tools understand. It can handle both suspend (.vmss) and snapshot (.vmsn) checkpoint state files as well as both monolithic and non-monolithic (separate .vmem file) encapsulation of checkpoint state data.

The vmss2core tool can produce core dump files for the Windows debugger (WinDbg) as well as for other operating systems. Please refer to given screenshot for more info.

For more info about usages of vmss2core tool, please refer to Debugging Virtual Machines with the Checkpoint to Core Tool

We need to take a snapshot of affected VM when it hangs, crashes, or otherwise display symptoms you are troubleshooting and then download the snapshot state file (.vmsn) and VM paging file (.vmem) which can later be converted to Windows memory dump file (.dmp) using vmss2core utility.

Steps:
1. Copy the vmss2core.exe utility to the same location where you downloaded the VM snapshot or suspended state files (.vmsn – in case of snapshot or .vmss for suspended state).

2. Open a command line and navigate to the location of the snapshot / suspended state files and execute the following command:

Here you would use the tool with OS‐specific options. For example, this command generates a memory.dmp file for the Windows debugger, WinDbg.

For Snapshot:
C:\folder>vmss2core.exe -W snapshot.vmsn [snapshot.vmem]

For Suspended state:
C:\folder>vmss2core.exe -W snapshot.vmss 

If the snapshot file is from a Windows 8 or Windows Server 2012 VM, use

C:\folder>vmss2core.exe -W8 snapshot.vmsn [snapshot.vmem]

Successful output of this should be a "memory.dmp" file suitable for use with WinDbg

Please note: VM paging file (.vmem file) may not be present depend on the state of VM.

Related VMware kb# 2003941 

That's it... :)

Fixing the error "vMotion is not enabled on the host of the Virtual Machine"

This is first time when I saw such error on a production host, which I know have everything properly configured (I mean, identical vmkernel port with vMotion enabled) and moreover vMotion was working for for this VMhost before.

When I saw this error at first, thought someone probably disabled vMotion inadvertently however on cross-checking found everything correct.

So, this is something happening due to probably some kind of technical glitch and to fix this we either need to reboot the host or simply disable/re-enable the vMotion in vmkernel port properties.

You can disable/re-enable the vMotion by editing respective vmkernel port but make sure you give it a few minutes before re-enabling.

That’s it…. 😊