How to spoof MAC address from inside Windows OS

Last day someone asked me, is there a way to spoof the MAC address of a NIC card on Physical host, here is the answer:

For a supported network card one can spoof the MAC address form inside the OS regardless its physical or virtual server(however this is not the preferred in case of VM, here is the preferred way:  assigning Static or mac address spoofing on a VM).

Follow these steps to do so:

• Go to NIC card's properties and select Configure,

• On Next Screen click on Advanced and then scroll and select Locally Administrated Address or something like that,   

here by default you will find it blank.

• Now select Value and inter the desired MAC address without : or -.

Once done click OK and this this done.

To verify the same open command prompt and use getmac or ipconfig etc to check the MAC address,

That's it.... :)

How to replicate MAC address during vNIC upgrade or P2V

When you upgrade a virtual machine's virtual network adapter you actually remove the old vNIC and then recreate a new vNIC and this new vNIC would have a new MAC address. In a different scenario during P2V we delete the old physical NICs and add new vNICs this also changes MAC address.
What if you have an application that is tied to the MAC address for licensing. In these situations you will have to replicate the old MAC address to the new vNIC. 

In case of vNIC upgrade it can be simply done from edit VM settings(can be done only for VMware reserved MAC address range i.e. 00:50:56:XX:YY:ZZ , where XX is a valid hexadecimal number between 00 and 3F, and YY and ZZ are valid hexadecimal numbers between 00 and FF) however to replicate the MAC address in case of P2V, you will have to edit the vmname.vmx file (as the physical server's NIC MAC address will be other than VMware reserved address range).

Note: If the MAC addressed is prefixed with 00:50:56 but it is outside of the [00-3F]:XX:XX range, it is still considered invalid and the VM will fail to power on.

Make sure you noted down the MAC address of old network card before removing it, there are many ways of checking the MAC address like using ipconfig or getmac command etc.

In case of vNIC upgrade: Open VM settings, select desired vNIC => Select manual and then change xx:xx:xx (xx:xx:xx is just to highlight by default its blank after 00:50:56:  ) with the required number.

Only in few cases like P2V where MAC address range is different from VMware approved range you will have edit the .vmx file as you can’t be replicate that MAC from VM settings.

• Shut down the virtual machine.
• Now right-click the virtual machine and click Remove from Inventory. 
• Download the .vmx configuration file for the virtual machine to your desktop using the Datastore Browser and open it using text editor 
• Now replace this line ethernet0.generatedAddress = "00:50:56:xx:xx:xx" with 

ethernetN.addressType = "static"
ethernetN.checkMACAddress = "false"
ethernetN.address = "xx:xx:xx:xx:xx:xx"

Where N is the vNIC number (Network Adapter 1 -> ethernet0 Network Adapter 2 -> ethernet1 etc.) and XX:XX:XX:XX:XX:XX is the new desired MAC address for the virtual machine.

Note: For more info, refer to VMware vSphere Documentation Center, KB# 219, 2035707

Other Thoughts: Yes of-course you can spoof the MAC address from inside the OS too but the problem is, in that case you will have to set MAC address change security policy to accept from vSwitch properties,

This is what that most of the VMware admins don't want to change.

That's it.... :)

VMware Announced vExpert 2015 Second Half

VMware has announced the list of 2015 Second Half vExperts.... I am very honored to be named VMware vExpert.... Congrats to all those named... :)

Here is the list of vExpert 2015 Second Half :  https://blogs.vmware.com/vmtn/2015/08/vexpert-2015-second-half-announcement-2.html 

VM failed to power on with the error: The object or item referred to could not be found

Last week I came across an issue where a powered off VM was showing a inaccessible and after fixing this issue when I tried to power on this VM, end up with this error,

"The object or item referred to could not be found"

Under tasks its showing as unregistering and then reregistering the VM (this task failed at 15%).

This was first time when I faced this issue so was amazed. Then I started looking about this error over web and luckily I found  VMware KB# 2047271 which is related to the error that I was getting.

As per this KB article this is a known issue affecting ESXi 5.0 and 5.1 and the work around of this issue is, one will have to rescan the datastores for the ESXi cluster. Rescanning datastores just for the host on which your VM is will not work, I am saying this as it didn’t work for me but you can give it a try too. Who knows it might work for you ;)

To rescan datastores for esxi host or cluster or datacenter, right click on it and select Rescan for Datastores…

Once the datastore rescan will complete, you will be able to poweron the VM.

That’s all…. J

Powered off VMs are greyed out and showing as not accessible

Few days ago I faced this issue where some powered off VMs were showing as not accessible in inventory.

When I checked vCenter alarm mails for that time around, found below listed lines in mail means this happened because of lost storage connectivity.

Alarm Definition:

([Event alarm expression: Lost Storage Connectivity] OR [Event alarm expression: Lost Storage Path Redundancy] OR [Event alarm expression: Degraded Storage Path Redundancy])

Path redundancy to storage device naa.6006019004cf12d00e0445e759755e598 degraded. Path vmhba2:C0:T0:L127 is down. Affected datastores: Unknown

Actually only VMs that were powered off at the time show up greyed out; all VMs that were running during the storage failure have been hanging, but they resumed properly after the LUN came back online. This issue is specific to 5.0, 5.1 hosts and related to NFS datastore but like in my case can occur with fiber channel storage too.

Fix: Same as orphan VM issue we can simply fix this inaccessible VM issue by removing affected VMs from inventory and then by re-adding the same (Go to datastore and browse the folder of affected VM, right click on vm configuration file (vm_name.vmx) and register it.

Caution: Before performing these steps, make a note of the datastore that the virtual machine resides on

One might have an issue with this procedure because vC will treat it as a new VM so the backup software, most probably you will have to configure backup again.

If you don’t want to remove/reregister the VM from/to inventory, you can do the same using vim-cmd command.

List the name of affected VM and connect to residing host over ssh using putty or DCUI,

Vim-cmd vmsvc/getallvms

Or to list all inaccessible VMs on this host,

vim-cmd vmsvc/getallvms >grep skip

Notedown the wids off affected VMs,

Vimcmd vmsvc/reload wid

This will reload the vm to inventory and fix the issue without removing and re-adding your VMs. Simply repeat the reload command for all the “skipped” VMs and your problem is solved.

BUT you will have to do this to one by one……what if you have multiple VMs with this issue…..there is another way where you can list/reload all inaccessible VMs at once using a simple one liner powershell script

#Get Inaccessible Virtual Machines

$VMs = Get-View -ViewType VirtualMachine | ?{$_.Runtime.ConnectionState -eq "invalid" -or $_.Runtime.ConnectionState -eq "inaccessible"} | select name,@{Name="GuestConnectionState";E={$_.Runtime.ConnectionState}}

write-host "---------------------------"

Write-host "Inaccessible VMs"

write-host "---------------------------"

$VMs

#Reload VMs into inventory

(Get-View -ViewType VirtualMachine) |?{$_.Runtime.ConnectionState -eq "invalid" -or $_.Runtime.ConnectionState -eq "inaccessible"} |%{$_.reload()}




Note: Restarting mgmt agents or disconnecting/reconnecting the Esxi host might also fix the issue.

Reference: Matt Vogt and Erik Zandboer's blogs.  

That's it.... :)

VM grayed out and showing as orphaned or invalid

In vCenter Server, you may find that you have a virtual machine that has an orphan designation or has become invalid. An orphan virtual machine is one that exists in the vCenter Server database but is no longer present on the ESX host. A virtual machine also shows as orphaned if it exists on a different ESX host than the ESX host expected by vCenter Server.

A virtual machine can show up as invalid or orphaned in these situations. To resolve these issues, see the troubleshooting steps provided for each situation:

• After a vMotion or VMware DRS migration
• After a VMware HA host failure occurs, or after the ESX host comes out of maintenance mode
• A virtual machine is deleted outside of vCenter Server
• vCenter Server is restarted while a migration is in progress
• Too many virtual machines are scheduled to be relocated at the same time
• Attempting to delete virtual machines when an ESX/ESXi host local disk (particularly the root partition) has become full
• Rebooting the host within 1 hour of moving or powering on virtual machines
• A .vmx file contains special characters or incomplete line item entries

Most of the time you can simply fix the issue by removing an orphaned virtual machine from the inventory and then re-adding it manually by re-registering. If this issue occurred due to unsuccessful host failover or when the virtual machine is unregistered directly on the host, you can fix the issue by migrating (of course cold) the affected VM to another host.

Reload all invalid virtual machines on a single host at one time

This script reloads all invalid virtual machines on a single host at once:

for a in $(vim-cmd vmsvc/getallvms 2>&1 |grep invalid |awk '{print $4}'|cut -d \' -f2);do vim-cmd vmsvc/reload $a;done

Referance: Vmware KB# 1003742, virten.net

That's it... :)

Virtual Machine startup - shutdown with ESXi host

Today we got a request from client to configure some specific site server virtual machines to automatically start and stop with host as there is frequent power cut on these sites so client wanted to make sure these VMs would come online as soon as the host will power on. 

These are individual hosts so no HA or DRS configured means no need to worry about a case where you migrated VM to another host and this configuration became irrelevant. 

Note: BTW automatic startup is not supported with vSphere HA.

Configuring a VM to startup - shutdown with ESXi host is very easy,

Using #C client,

• In the vSphere Client inventory, select the host where the virtual machine is located and click the Configuration tab.
• Under Software, click Virtual Machine Startup/Shutdown and click Properties.
• Select Allow virtual machines to start and stop automatically with the system, then Configure the startup and shutdown behavior.

 for detailed info about available optional settings, checkout VMware vSphere Documentation Center =>> Managing Virtual Machines section.

Using Web Client,

• In the vSphere Web Client, navigate to the host where the virtual machine is located.
• Select Manage > Settings.

• Under Virtual Machines, select VM Startup/Shutdown and click Edit, edit settings dialog box will open,
• Select Allow virtual machines to start and stop automatically with the system, then Configure the startup and shutdown behavior options.

That't it... :)

vNIC upgrade issue: IP address already assigned to another adapter

Last week when I was upgrading VMware Tools and vNICs of few virtual machines, on two server 2003 machines faced this issue where I was not able to assign  previously assigned IP to newly added vNIC card, ended up with this error,

Some people say what a big deal its a very common post P2V or vNIC upgrade issue we face, agreed.

But I was surprised because even after following best practice during the network card upgrade why I still faced this issue. Here is what I did during the upgrade activity:

• Updated VMware Tools and rebooted the VM 
• Noted down the IP address then uninstalled the NIC card from device manager and powered off the VM 
• Removed the old e1000/vmxnet vNIC and added VMXNET3 network card then powered on VM
• Now tried to assign the previously assigned IP address to newly added vNIC

As I already removed the previous vNIC so was supposed to not face this issue anyways now come to fix. 

Most of the time we can fix the issue by simply running this env variable command 

set devmgr_show_nonpresent_devices=1, (leave the command prompt open)

and then delete the ghosted network adapters from Device manager (view=> show hidden devices),

 Now close the command prompt.

But what if you are still not seeing any ghosted network adapter in device manager ((view=> show hidden devices is checked).

If the above listed command does not work (a possibility in Windows Server 2000 and 2003), you may need to add this environment variable devmgr_show_nonpresent_devices to Windows and set its value to 1 in order view/delete the ghosted network adapter from device manager, to do so,

• Right-click the My Computer desktop icon and choose Properties.
• Click the Advanced tab and select Environment Variables.

• In the System variables section, click New.

• Set the Variable name to devmgr_show_nonpresent_devices and set the Variable value to 1 to enable the parameter.

• Click OK to add the variable to Windows.

Now open the Device Manager, Click View > Show Hidden Devices.

1. Expand the Network Adapters tree 
2. Right-click the dimmed network adapter, then click Uninstall.
3. Once all of the grayed out NICs are uninstalled, assign the IP address to the virtual NIC.

For more info refer to VMware KB# 1179 & Microsoft KB# 269155.

Update, 05/10/2015: Last day when I was working on a Windows Server 2008R2 machine, found one may assign an already assigned IP to new vNIC even without removing the old ghosted vNIC. As the warning message suggests 

Once you would click Yes, then system will remove the conflicted static IP configuration from ghosted NIC card.
But if you had assigned a name to vNIC and want to keep that same name for new network adaptor then you will have to first remove the old hidden vNIC.

The above process works fine in case of MS server 2003 but its slightly different for Server 2008 and that might be very frustrating if you are not aware.

The difference is, you need to start the command pormpt in elevated mode (run as Administrator) and after running the SET DEVMGR_SHOW_NONPRESENT_DEVICES=1 cmd you also need to open the device manager from same cmd by running START DEVMGMT.MSC command.

Other steps are same.

 

That's it.... :)