Inconsistent LUN mapping related issues on ESXi hosts

Lately came across this issue, where for some reason storage team unmapped and re-mapped few RDM LUNs to VM host group (from Storage array side) and now the respective RDM disks connected to VMs get disappeared.
We had already re-scanned the hosts to storage change and Luns were showing as mounted on all the hosts and after spending two hours with VMware support we had also rebooted the host but that didn't make any difference.

Finally when we rebooted the cluster nodes then I found this has something to do with consistent mapping of rdm Luns across VM hosts (where the cluster nodes residing).

In order to check if a LUN is consistently mapped on all VM hosts in cluster, one need to have a look at Lun's canonical name's (naa.id) corresponding vmd.id

One can check the naa.id's corresponding vml.id by running following cmd  on host (over ssh, using putty),
esxcli storage core device list -d naa.id

So, if the naa.id is naa.60060480000190104063533030353445 then the command would be,

esxcli storage core device list -d naa.60060480000190104063533030353445

For example,  vml.02000500006006048000019010406353303035344553594d4d4554

One need to look at the fifth and sixth digits (see highlighted) of vml.id, this is hexadecimal number which represents the LUN number. On converting to decimal it should match to actual Lun number.

Now to fix this issue what we can do is,  remove the affected RDM disk from the both nodes and then delete the RDM pointer file from Datastore (this doesn’t affect your actual data on LUN). Now after re-scanning the hosts for Datastores, re-add the LUN as RDM drive on both nodes. Now you would be able to power on the affected node.

If due to any reason above doesn’t work then as above after removing the affected RDM drives from both nodes, follow these steps,

1. Note the NAA_ID of the LUN.
2. Detach RDM using vSphere client.
3. Un-present  the LUN from host on storage array. 
4. Rescan host storage. 
5. Remove LUN from detached list using these commands:
   
   #esxcli storage core device detached list
   #esxcli storage core device detached remove -d naa.id
6. Rescan the host storage. 
7. Re-present LUN to host. 
8. Now again rescan the hosts for datastores

Now cross check the vml.id on hosts and it should be same and after adding the RDM drive on nodes you will be able to power on the VM nodes.

Note: If the LUN has been flagged as perennially reserved, this can prevent the removal from succeeding and step 5 would fail.

Run this command to remove the flag:

#esxcli storage core device setconfig -d naa.id --perennially-reserved=false

Now the command to remove the device should work.

# esxcli storage core device detached remove -d naa.id

I had faced a related issue in past and discussed about that in following post, 

After unexpacted host reboot, Powering on a RDM attached virtual machine fails with the error: Incompatible device backing specified for device '0

That's it... :)

AWS Public IP vs Elastic IP and how can we assign one to EC2 instance

In my previous post I mentioned that in order to make an EC2 instance internet accessible, it should have a Public or Elastic IP assigned. Here anyone new to AWS may wonder that, what is this Elastic IP and How its different from Public IP.

In this post we will discuss about the similarities and differences between these two and how one can assign a Public or Elastic IP to an EC2 instance.

If you are looking from a functional point of view then they both are publicly routable IP addresses and can be used to connect your instance to internet but are different how they persist and the way you can assign one to your instance.

A public IP address is assigned to your instance from Amazon's pool of public IPv4 addresses, and is not associated with your AWS account. When a public IP address is disassociated from your instance, it is released back into the public IPv4 address pool, and you cannot reuse it.

If we put it in simple word then, Public IP addresses are dynamic, which means if you stop/start your instance you get reassigned a new public IP however it would persist if you just reboot the ec2 instance.

Public IP addresses are free and you will not be charged anything for using them.

An Elastic IP address is a static public IPv4 address, designed for dynamic cloud computing. If your instance does not have a public IPv4 address, you can associate an Elastic IP address with your instance to enable communication with the internet.

An Elastic IP address is associated with your AWS account and with it, you can mask the failure of an instance or software by rapidly remapping the address to another instance in your account.

While your instance is running, you are not charged for one Elastic IP address associated with the instance, but you are charged for any additional Elastic IP addresses which is not in use.

Now let’s look at the difference between these two IP types.

1. Elastic IPs are assigned to AWS accounts which you can attach to instances. Public IPs assigned to instances directly.

2. You cannot manually attach or detach public IP from the instance. It’s auto allocated from the pool. Elastic IP can be manually attached and detach from the instance.

3. When an instance is stopped and started again, public IP gets changed. But if the instance is assigned with elastic IP, it will remain the same even if the instance is stopped and started again.

4. If elastic IP is allocated to your account and not in use then you will be charged for it on an hourly basis.

5. Public IP released once your instance is stopped so no question of getting charged for not using it.

6. You won’t be able to re-use same public IP since its allocated from free IP pool. You can always re-use, re-attach elastic IP to other instance when it is released from current instance.

7. You can have maximum 5 elastic IP to your account per region. But, you can have as many public IPs as EC2 instances you spin up.

8. You can have either of them for an instance. If you assign elastic IP to instance then its currently assigned public IP will be released to the free pool.

How would be assign Public or Elastic IP to an instance: 

Public IP: It can be assigned to an instance only during the instance creation time and there are two ways of doing that.

• Edit the Subnet setting and enable Auto-assign Public IP to any EC2 instanced launched in this Subnet.

To do so, From AWS Console => Under Networking & Content Delivery, Select VPC => Now Click on Subnet Tab => Select the intended Subnet and either right Click or from Actions => Select Modify auto-assign IP address

That will open the following Modify auto-assign IP address pop-up, now as shown enable auto-assigning Public IP address.

Now any EC2 instance which would be launched in this subnet would have a public IP assigned.

• You can also assign the Public IP during EC2 instance launch time, you can also alter the default public IP assignment in a subnet from here.

Whatever you select here would over right the default IP assignment settings.

Elastic IP: We need to allocate the Elastic IP address to our AWS account before making use of it.

You can go to Elascic IP windows either from EC2 instance or VPC Dashboard, once you are there then => Select Elastic IP address => Allocate new address

There is not much to discuss here, once you would click on allocate in next screen, It would allocate you an Elastic IP.

Now if you want to assign this IP to any instance, just select it and either click on Actions or Right click on it and Select Allocate address.

That would open the following Associate Address window, from here you can select the intended EC2 instance or specific network interface.

Once select the intended instance, the elastic IP would get associated with the selected instance.

Note: As mentioned in above screenshot, if you associate an Elastic IP address to an EC2 instance which already has a public IP assigned, the public IP is released.

That's it ... :)

Launching your first AWS EC2 instance and making it internet accessible

I assume this is your first EC2 instance and you just have the default VPC with default Security groups, Network Access list and Routing table etc.

As this EC2 is being created in default VPC, which by default has an Internet Gateway attached with a route specified to internet gateway in attached route table so, one only need to complete following steps to make it internet accessible,

• Create this EC2 instance with a Public IP

• If the instance was not created with a public IP then create an Elastic IP and attach it to the EC2 instance

• Identify the attached security group and create a custom inbound rule to allow traffic (RDP and ICMP etc) from an specific IP/Network or from anywhere (0.0.0.0/0)

Here you don’t need to create any outbound rule because Security groups are state full and on the other hand Network access lists by default allow all the traffic.

Note: If you didn’t select a security group during ec2 launch then the ec2 instance would have a new security group name Launch-Wizard-x created and attached, then you need to create custom inbound rules to allow RDP or ICMP traffic for this group.

If required, you can change the attached security group for an instance from here EC2 Dashboard => Network & Security =>Network interfaces.

That's it... :)

Amazon Web Services: A short description of some of the AWS services

As I am planning to sit in my next AWS Certification exam which is Architect Professional in next few months so, thought of starting from scratch and make a note of all common AWS services with their short description (Actually at times I find it hard to recall what service is there for a particular use case so...from next time, I can have a look here).

Amazon Web Services offers a broad set of global cloud-based products including compute, storage, databases, analytics, networking, mobile, developer tools, management tools, IoT, security and enterprise applications. These services help organizations move faster, lower IT costs, and scale.

Compute Services :-

Amazon EC2 : Virtual Servers in the Cloud

Amazon EC2 Auto Scaling : Scale Compute Capacity to Meet Demand

Amazon Container Service : Run and Manage Docker Containers

Amazon Lightsail : Launch and Manage Virtual Private Servers

AWS Batch : Run Batch Jobs at any Scale

AWS Elastic Beanstalk : Run and Manage Web Apps

AWS Lambda : Run your code in Response to events

VMware Cloud on AWS : Build a Hybrid Cloud without Custom Hardware

Storage Services :-

Amazon S3 : Scalable Storage in the Cloud

Amazon EBS : Block Storage for EC2

Amazon EFS : Managed File Storage for EC2

Amazon Glacier : Low cost Archive Storage in Cloud

AWS Storage Gateway : Hybrid Storage Integration

Database Services :-

Amazon RDS :  Managed Relational Database Service For MySQL, PostgreSQL, Oracle, SQL Server and MariaDB

Amazon Aurora : High Performance Relational Database

Amazon DynamoDB : Managed NoSQL Database

Amazon Redshift : Fast, Simple, Cost-effective Data Warehousing

Amazon ElastiCache : In-memory Data Store and Cache

Amazon Neptune :  Fully Managed Graph Database Service

Migration Services :-

AWS Database Migration Service : Migrate Database with Minimal Downtime

AWS Application Discovery Service : Discover On-Premises Application to Streamline Migration

AWS Server Migration Service : Migrate On-Premises Servers to AWS

AWS Snowball : Petabyte-scale Data Transport

AWS Snowball Edge : Petabyte-scale Data Transport with On-board Compute

AWS Snowmobile : Exabyte-scale Data Transport

AWS Migration Hub : Track Migrations from a single place

Networking and Content Delivery Services :-

Amazon VPC : Isolated Cloud Resources

Amazon Route 53 : Scalable Domain Name System

AWS Direct Connect : Dedicated Network Connection to AWS

AWS CloudFront : Global Content Delivery Network

Elastic Load Balancing : High Scale Load Balancing

Amazon API Gateway : Build, Deploy and Manage APIs

Management Tools :-

AWS CloudWatch : Monitor Resources and Application

AWS Auto Scaling : Scale Multiple Resources to Meet Demand

AWS CloudFormation : Create and Manage Resources from Templates

AWS CloudTrail : Track User Activity and API Usage

AWS Config : Track Resources and Inventory

AWS OpsWorks : Automate Operations With Chef and Puppet

AWS Service Catalog : Create and Manage Standardized Products

AWS System Manager : Gain Operational Insight and Track Action

AWS Trusted Advisor : Optimize Performance and Security

AWS Personal Health Dashboard : Personalized View of AWS Service Health

Analytics :-

Amazon EMR : Hosted Hadoop Framework

Amazon Kinesis : Work With Realtime Streaming Data

Amazon Redshift : Simple, Fast, Cost-effective Data Warehousing

Amazon CloudSearch :  Managed Search Service

Amazon Elasticsearch Service :  Run and Scale Elasticsearch Cluster

Amazon Quicksight : Fast Business Analytics Services

AWS Data Pipeline : Orchestration Service for Periodic, Data-driven Workflow

AWS Glue : Prepare and Load Data

Amazon Athena : Query Data in S3 using SQL

Security, Identity & Compliance :-

AWS Identity & Access Management : Manage User Access and Access Keys

AWS Single Sign-On : Aloud SSO Service

AWS Organizations : Policy-based Management for multiple AWS accounts

AWS Key Management Service : Managed Creation and Control Of Encryption Keys

Amazon Inspector : Analyzed Application Security

AWS Shield : DDoS Protection

AWS Guard Duty : Managed Threat Detection Service

AWS WAF : Filter Malicious Web Traffic

......................................................................

...........................................................................

................................................................................

Reference: https://aws.amazon.com/products/

My previous AWS exam related posts can be fond here,

AWS Certified Solution Architect exam experience and few exam related tips

AWS Certified SysOps Administrator - Associate exam experience and few exam related tips

Hope this would be useful for others as well...That's it for now :)

VMware vExpert 2018 award Announced

After lots of delay VMware finally announced the list of vExpert 2018…. I am very honored to be named a VMware vExpert again, this is my fourth vExpert award…..

Congrats to all those who named as a 

Here is the full list of vExpert 2018... https://vexpert.vmware.com/directory 

vExpert 2018 Announcement on VMTN Blog,

https://blogs.vmware.com/vmtn/2018/03/vexpert-2018-award-announcement.html

That's it... :)

Unable to power-on a VM in nested ESXi, hanging at some random point with a question in the events

I would say this is the most weird issue I ever faced, it took me so long to find a solution so thought of making a note of it.

My setup is all nested, VMware workstation on my Lab Windows PC and then ESXi, vCenter and everything else as VMs running on workstation.

After building a vSphere 6.5 lab environment, when I deployed my first VM and tried to power it on it got stuck at a some point, after waiting for few minutes when checked the task and events, found following info event there,

"Running VMware ESX in a virtual machine will result in degraded performance. Do you want to continue"

Then I created few other VMs but they all got stuck at some random point during power-on operation and the same event was there in events.

However there is no way to answer the question, nothing in VM summary and no pop-up nothing.

A quick search about the issue lead me to VMwareb# 2108739 however adding msg.autoAnswer = "FALSE" in host .vmx file or /etc/vmware/config didn't make any difference.

Initially I thought this is something related to ESXi 6.5 so, just to check deployed another VM with ESXi 6.0 but even on nested ESXi 6.0, VM power-on operation stuck at random point.

Earlier, I had a working vSphere 5.5 nested lab on this system so, thought of checking with ESXi 5.5 but to my surprise VM power-on operation got stuck with same "Running VMware ESX in a virtual machine will result in degraded performance. Do you want to continue" question (without any way to answer it) for ESXi 5.5 too.

Then I thought this is something related to VMware workstation so, downgraded it from version 14 Pro to 12 Pro but there was no difference.

This is when I started looking at other things and eventually found that its my antivirus which was causing this issue.....Avast Antivirus...... enabling or disabling the features in the software did nothing however uninstalling Avast worked like a charm and now everything working again as expected.

Hope this would help...

That's it ...:)

How to Check, Start, Stop or Restart Windows version of vCenter 6.x Services

Today while working on something I rebooted my vCenter server and post reboot when tried to connect to vCenter using Web client, end up with following error,

This error is something which usually occurs when there there is some issue with vCenter Web client, probably the service is set to manual or somehow got stopped.

Ideally Web client service should have startup type set as Automatic and in start state.

To verify this, opened Services and to my surprise couldn't find the vCenter Web Client Service listed there, even vCenter service was not there (ideally this shouldn't be the case).

So, here how would you check the status of Web Client Service or for any other service not listed in Services.

For Windows version of vCenter, to list the vCenter Server and/or Platform Services Controller services,

• Open command prompt as Administrator and change the directory to C:\Program Files\VMware\vCenter Server\bin (or any other relevant path, if you didn't install vCenter on default location)

  Now we will use Service-control utility, available here to check the status of service or perform a related operation on it.

      To view the available option, run service-control --help

• Run following command to list vCenter Server and/or Platform Services Controller services,

          service-control --list-services

• Run this command to view the current status of all vCenter Server and/or Platform Services Controller services: 

    service-control --status

     To check the status of individual services, use this,

      For Web client Service:

    service-control --status vsphere-client

  For vCenter service:

         service-control --status vpxd

    

 From here, you can start, stop any of vCenter Server and/or Platform Services Controller   services using below cmd,

service-control --start 'Service Name'

Or to start all services,

service-control --start --all

Note:To perform a dry run of the command, add the option --dry-run to the command, doing so will display what actions the command will run without executing the actions. 

Hope this would be helpful.

Reference: VMware kb# 2121043 & 2109881
Related Post: VMware vCenter Server 6.x Appliance services: how to find service status or start/stop

That's it... :)