VMware Sample Exchange beta Site

Today I heard from vExpert program team about VMware Sample Exchange beta Site and spent some time on this site, as the site is its initial stage so there is not much script samples available yet however the idea seems very promising and useful.

On Sample Exchange Site one can find code and script samples from VMware as well as from members of the VMware community. Here we can discover, contribute, and request samples in PowerShell, Python, Ruby, Java, and many other languages. Sample Exchange is currently in Beta and features content from Alan Renouf (@alanrenouf), William Lam (@lamw), and other VMware experts. 

The beta is for a period of 60 days and in this period only vExperts can contribute/request for sample code/scripts and Non-vExperts can only browse and download.

Useful URLs: 

• VMware Sample Exchange Site - http://developercenter.vmware.com/samples

• Sample Exchange Community on VMTN - https://communities.vmware.com/community/vmtn/resources/se 

That's it... :)

Server has a weak ephemeral Diffie-Hellman public key error in Chrome/Firefox

I believe anyone who is using vSphere Web client on version 5.1 would be aware about this error, we get this error when try to connect to web client or any other site having certain SSL Ciphers using latest versions of Chrome/Mozila (so far I didn't see this issue with ie),

Note:- This is a known issue affected the vSphere Web Client 5.1,  it is resolved in vSphere Web Client 5.1 Update 3e and later
This issue occurs due to changes to the web browser containing a fix to combat an unrelated vulnerability that consequently disables certain SSL Ciphers.

When I was looking for how to avoid this for web client or any other site giving this error, I came across the thread about the related issue on Google Chrome Help Forum and the summary is, so far Chrome itself doesn't have any option to disable related setting to allow the sites having relatively week security.

If a secure website gets the error ERR_SSL_WEAK_SERVER_EPHEMERAL_DH_KEY, it means the website is trying to set up a secure connection, but it is actually IN-secure because the SSL/TLS uses a Diffie-Hellman group size smaller than 1024-bit.

This is the problem in the Logjam vulnerability, which affects both browsers and servers:   https://weakdh.org 

In this case, the website/webserver needs to be fixed.  Google Chrome won't use insecure connections in order to protect your privacy.

In my case I am using self signed certificate instead of certificate authority signed certificate.

Resolution:- Google Chrome:- As I earlier said the there is no option available within Chrome to enable you to access less secure sites over https however as a way around we can use IE Tab Chrome Extension it will allows us to open vSphere web client within Chrome.

To use this extension, first go to Chrome Web Store and add IE Tab extension to chrome, now go to your url, you will again get the "Server has a weak ephemeral Diffie-Hellman public key error" Now all you have to do is click on the IE Tab icon which you will find in the right corner of the Chrome window (Highlighted in Blue),

And once you would click on IE Tab icon,

Though it's not an official fix, it still works and would allow you view the web pages without any issues.

In Mozilla Firefox we have an option to disable it by going to following url,

about:config

Here in this config page, you will find a list of boolean entries. Search for below two entries,

security.ssl3.dhe_rsa_aes_128_sha

security.ssl3.dhe_rsa_aes_256_sha 

By default, these are set to TRUE. But you have to set them to FALSE in order to allow the less secured pages.

Reference:  kb# 2125607, Senthil Kumar Murugesan's blog.

That's it... :)

Open VM console error: The VMRC console has disconnected...attempting to reconnect

This was second time when I came across this issue where I was not able to open the VM console, when tried to open VM console, end up with an error i.e, "The VMRC console has disconnected... attempting to reconnect"

And again it took me some time to figure out the issue so thought of writing a post about this issue.

This is nothing but just a local vSphere clients issue(might be an application like antivirus is blocking application functionality) and in order to fix it you just need to kill the vSphere client related services. Open Task manager and look for any vmware-vmrc.exe process(es), Select it, right click and now select Kill Process Tree.

Once you have killed all VMware vSphere client related process ((vmware-vmrc.exe/vmware-remotemks.exe/vpxclient.exe), Now open up the vSphere Client, console should now work.

A reboot of VMware vSphere client system will also fix the issue but that's not possible every time as one might have some other applications running on this host but if above doesn't work then you will have to give this a try or you may also upgrade/reinstall the vSphere Client.

Reference: kb# 2050470, 2032016

That's it... :)

ESXi installation types: Embedded/ Installable, how would you determine

As you know ESXi is the unified version of the VMware hypervisor however during installation on the basis of size of destination media you have, its installation can be categorized as one of these types:  

§  Embedded : Installed in the attached SD card or USB

§  Installable : installed on a local hard drive

There is one more type, that's,

§  PXE: used in the AutoDeploy environment

During the ESXi installation process you will never be asked whether you want to install in embedded or installable mode. It solely depends on the type and size of your target installation media:

·         If you install ESXi on a USB key drive or SD card then you will always end up with ESXi embedded.

·         If you install ESXi on a hard disk (or iSCSI/SAN/FCoE partition) that has a size of at least 5 GB then you will end up with ESXi installable.

·         If the installation target media (no matter what type) is smaller than 5 GB then you will end up with ESXi embedded.

Now you wonder, in case of PXE, how would you determine if its Installable or Embedded version? The destination media is not enough because you can install ESXi also over the vendor’s SD card used for the embedded versions. Here is the answer,

To determine the type of ESXi installation:

1. Connect to the host via SSH.
2. Run this command: # esxcfg-info -e

You see an output similar to:  boot type: visor-thin 

You can determine the ESXi type based on the output of this command.

For example:

ü  visor-thin indicates an installable deployment

ü  visor-usb indicates an embedded deployment

ü  visor-pxe indicates a PXE deployment

Note: For ESXi embedded it is a good practice and a recommendation by VMware to create a persistent scratch location to store log files otherwise you would lose them during host reboot.

Reference KB# 2014558, Andreas Peetz's blog post. 

That's it... :)

How to upgrade firmware of a HP ProLiant G9 server (upgrading firmware from version 2.20 to 2.30)

As I posted earlier, we was facing embedded Flash/SD-Card related issues: Lost connectivity to the device mpx.vmhba32:C0:T0:L0 backing the boot file system error on vSphere client host summary page, these hosts are running on HP ProLiant G9 servers. As this was second time when we saw this error so instead of fixing it myself, contacted HP support to find out the root cause of the same. The response of hp support was as expected, 

Response from hp support:  That version 2.20 has been removed from our site due to it causing issues with server components, including the embedded flash cards. . The new iLO firmware 2.22 addresses/fixes issues with the embedded cards disconnecting.

In our further discussion they suggested us for firmware upgrade from 2.20 to 2.22 or to the latest available version 2.30.

There are many ways to upgrade firmware, few are as follows:

• Upgrading firmware directly from iLO using firmware update bin file. 

First download the firmware version 2.30  or any other version from HP site.

Getting the firmware setup .bin file is little tricky.To find the .bin file, first extract the firmware setup zip, there you would find an executable file now again extract the executable file by using 7-Zip/WinRAR etc. By this way you will get the firmware upgrade .bin file.

Now connect to iLO, G9 servers having iLO version 4,

Go to Administration => Firmware => form here one can upgrade the firmware of a HP server by uploading the firmware upgrade .bin file.

iLO will reboot during the firmware upgrade and Server reboot is not required.

• If your is running Esxi on it then you can even upgrade the firmware directly from DCUI or using SSH.

Note: If you would extract the downloaded firmware setup zip folder, you would get a Readme file, this file would have instructions to upgrade firmware from within esxi console.

or Follow these steps to upgrade HP server’s firmware from inside the Esxi console:

Put the intended host in maintenance mode however most of the case the reboot is not required but just in case. Now Open putty and connect to host using root credentials.

Copy the downloaded firmware zip (CPxxxxxx.zip) file to a temp or any directory and then browse to that directory.

• Now from the same directory, unzip the Smart Component:

                unzip CPxxxxxx.zip

•  To ensure CPxxxxxx.vmexe is executable, execute the commmand:

                 chmod +x CPxxxxxx.vmexe

•  And then to finally upgrade the firmware run this command,

              ./CPxxxxxx.vmexe

Now follow the directions given by the Smart Component.

Once the firmware upgrade would complete, iLO will reboot and If instructed, reboot your system for the firmware update to take effect however it didn't ask us for reboot but anyways we rebooted the server.

• If you want to upgrade the firmware as well as other server component, then use HP SPP (Service pack for ProLiant), using it you can upgrade individual server component/driver or everything.

What you need to do is, just download the latest/intended HP SPP ISO file from HP Support Site, Now open iLO remote connection and mount SPP ISO as virtual CD/DVD and reboot the server.

During server boot choose boot from CD/DVD.

Once the Server would boot up using the SPP ISO, you get two options to upgrade server components: Automatic and Interactive, chose one as per your convenience. 

I personally prefer Introspective mode, as using this mode I can upgrade individual components and if any upgrade failed one would get the info right there. 

That’s it... :)

Remote connection failed with the error, outdated entry in the DNS cache

Last day I came across an issue where an user was not able to take remote of a windows VM using the server name however he was able to take remote of it using IP. He was getting an outdated DNS entry related error when taking RDP using server name, 

In first look it looks like a local dns cache related error so cleared the dns cache of his system by running ipconfig /flushdns and tried to connect again but end up with same error. Then I tried to connect to affected server from my system but as earlier got the login screen and after entering my credential as soon as I hit enter, end up with the same “…..This could be caused by an outdated entry in the DNS cache….” Error. Then I tried to login using the IP address of the server and was able to login without any issue.

After some investigation I noticed the affected system's time is different than the DC means not in sync with DC/DNS server so I restarted the Window Time Service and ask the user to try to login and this time user was able to login without any issue.

One can start the windows service either from services manager (Run => services.msc) or from command prompt. To restart the windows service from cmd run below commands.

net stop w32time

net start w32time 

Once the Windows time service would restart system  will sync the time with DC again and you will be able to login using server name.

Note: this issue is related to Windows time sync with DC so I'm sure it's not specific/limited to VM. 

That's It... :)

Lost connectivity to the device mpx.vmhba32:C0:T0:L0 backing the boot filesystem : HP ProLiant Gen9 Server

Last day when I logged in to vCenter I noticed one of the host with warning icon and upon checking in summary tab found this warning,

This warning message is indicating the Embedded Flash/SD-Card (esxi embedded install) is no longer available to the ESXi host.  As this a HP ProLiant server so logged into iLO, checked into diagnostics and found the SD-Card status ok then taken a look into ilo logs and found SD-card was restarted recently.

The good news is that the whole ESXi OS loads into memory so there was no outage for the VMs and once the connectivity would restore the host can access the storage again. The bad news is that the error did not clear automatically and as no one likes to see errors/warning in their production environment so I needed to find a solution to this issue.

The simplest solution of this issue is to put the host in maintenance mode and restart the management agents. One can do this by two ways, either connect to the host using ssh and run below commands,

/etc/init.d/hostd restart

/etc/init.d/vpxa restart

Or alternatively connect to the host using iLO, establish a remote connection, login to DCUI and restart the management agents.

                                       

Once the managements agents restart will complete, vCenter will show the host back in a normal state.

Note: There might be cases where SD-Card having issues due to buggy firmware and in order to fix the issue you may need to upgrade or downgrade the firmware.

we are at firmware version 2.20 and as per various forums this version have SD-Card related bug and that was supposedly fixed in firmware version 2.22, as version 2.30 is also available so one may upgrade to one of these versions of firmware.

Other Scenario: What if SD-Card is failed, you can try to remove and reattach the SD-Card but if it still doesn’t come online then you need to call the server vender for its replacement. 

But if SD-Card is bad, migrate all VMs to other hosts then put the host in maintenance mode and take backup of host configuration. Now shut down the host and after replacing the flash drive reinstall the esxi (As the host will not come up after reboot), once the host comes up, configure the management network and VLANs then restore the host configuration.

Reference: Daniel's blog and discussion on other forums.

Update, 05/11/2015:- This week we faced the same issue again so instead of fixing it myself contacted HP support and they confirmed the issue is with firmware version 2.20 that we have on these G9 server.

Response from hp support:  That version 2.20 has been removed from our site due to it causing issues with server components, including the embedded flash cards. . The new iLO firmware 2.22 addresses/fixes issues with the embedded cards disconnecting.

  

And hp support further suggested us to upgrade the firmware of G9 servers to most recent version 2.30.

That’s it… :)

vCenter Server 5.5 installation/upgrade stuck at Orchestrator installation screen

Last day when I was re-installing vCenter server in our test environment, came across this issue where vCenter install process stuck at Orchestrator installation screen for indefinite time (waited for around two hour). 

Initially I thought there might be something wrong with my machine as I had not rebooted the system after vCenter uninstall so I clicked on cancel to end the installation without success then I had to kill the installation process from task manager.
Once the installation terminated, I uninstalled the SSO, inventory service and Orchestrator (yes it was there in add remove program) and rebooted the system. After reboot installed SSO, inventory service and then initiated the vCenter installation again but to my surprise, vCenter installation stuck on same Installing Orchestrator screen again. Then I started Googling about this issue and found a VMware KB article about the same.
As per KB# 2051216,

This is a known issue affecting vCenter Server 5.1 Update 1 and vCenter Server 5.5

To work around this issue:

1. In the Windows Task Manager, locate the vRealize Orchestrator (formerly known as VMware vCenter Orchestrator) processes. For example, you may see processes similar to:
   
   vCenterOrchestrator.exe *32 InstallAnywhere Self-Extractor
   vCenterOrchestrator.exe *32 LaunchAnywhere
2. End the process tree for the vCenterOrchestrator.exe *32 LaunchAnywhere process.
   
   The vCenter Server installer should now complete successfully.

Note: If you are a user of vRealize Orchestrator (formerly known as VMware vCenter Orchestrator), you can complete the configuration manually through the vRealize Orchestrator configuration service.

Related KB article# 2045613.

That's it... :)