AWS Single Sign-on and Azure AD Application Certification rotation

In this quick post would discuss the process and steps involved in rotating the expiring Azure AD application certification configured for AWS SSO login. 

This is applicable where you have your AWS account SSO configured with Azure Active Directory and the associated application password is about to expire or maybe already expired.

Before you start, make sure to have the appropriate AWS IAM and Azure AD permission or involve the teams having the required access to create an application certificate (in Azure) and rotate the same in AWS.

Now, login to AWS and take the backup of currently used metadata.

1. Login to AWS => Go to IAM => Click on Dashboard or from the IAM menu, click on Identity Provider
2. Click on Azure AD => From Metadata Document section, Download the current metadata file for backup purpose

Now Login to Azure,

1. Go to Azure Active Directory => Select Enterprise applications from left menu options
2. From the Enterprise applications section, Select the correct AWS Application used for SSO 
3. No on the AWS Application screen, go to Single Sign-on option => SAML Signing Certificate and click Edit
   
4. On SAML Signing Certificate Page, Create a new Certificate, Save and mark it as Active, close the window
   
5. Now on SAML Signing Certificate Page, verify the certificate Expiry date and Download the Federation Metadata XML
   
   
   
   
6. Go to AWS account IAM Identity Provider Section, Steps are mentioned above
7. Within the Metadata Document section, this time Click on Replace Metadata, on pop-up window Type replace and Click on Replace tab. Just in case if you didn't download the current metadata file earlier, do that so just in case of any issue you could revert
   
8. Now browse and select the Federation Metadata XML file downloaded after Azure AD application certificate rotation and click open
9. It would take the next few seconds and you are done.
10. Test your AWS Single Sign-on URL, you can also perform the testing from within the Azure Application SAML bases Single sign-on page.

Note: If you are using an AD account to replace the AWS Identity provider Metadata then make sure to log in prior to marking the newly created Azure application certificate active. Also, don't refresh the AWS login page until you replace the metadata.

To avoid this, simply use your AWS root account ;)

Related Demo: 

That's it, thanks :)

AWS Glue Vs. Azure Data Factory : Similarities and Differences.

In today’s world emergence of PaaS services have made end user life easy in building, maintaining and managing infrastructure however selecting the one suitable for need is a tough and challenging task. We often tend to select hybrid cloud solution for our customers thus providing them the cost efficient solutions with cutting edge technologies.

The fundamental building block of any company is DATA  , without which no organization can think of survival. But to store and analyze this Data is the traditional approach of warehouse is not fit well because of many reasons. It could be increasing cost or infrastructure or over head of management ,but it does not fit well today.

The other alternative we have is Cloud , be it AWS / Azure /Google or any other. Each of these cloud offer different solutions to problems that we have. But fundamental Question remain same , which cloud to use and why.

Take Data analytics itself , For Running ETL jobs both AWS and Azure offer some solutions , but as architect we need to deeply understand the similarity and differences between two , before suggesting that to customer.

I am here highlighting the some fundamentals similarities and differences between two technologies  hoping that it might help the individuals who need to make solutions for customers .

Similar Features for two services 

Attribute	AWS Glue	Data Factory
Fully Managed, Server-less ETL engines	Yes	Yes
Data ingestion as both structured as well as unstructured data.	Yes	Yes
Auto generation of code	Yes	Yes
Underlying technology stack: Spark	Yes	Yes
Trigger type can be manual as well as automatic	Yes	Yes
Enable you to focus on building business logic and data transformation	Yes	Yes
Perform data cleaning, transformation and aggregation	Yes	Yes
Connects to data warehouses. Data lakes?	Yes, Support data to and from Redshift	Yes : Support  in and out from SQL DW
Transparent Pricing	Yes	Yes
Support SLAs	Yes	Yes
Ability for customers to add new data sources	Developers can write custom Scala or Python code and import custom libraries and Jar files into Glue ETL jobs to access data sources not natively supported by AWS Glue.	Yes

Differences between these two services

Attributes	AWS Glue	Data Factory
Main Focus of service	ETL, data catalog	ETL
Database replication	Full table; incremental via change data capture through AWS Database Migration Service (DMS)	Full table; incremental via custom SELECT query
SaaS sources	None	About 20, with several more in preview
Compliance, governance, and security certifications	HIPAA, GDPR	HIPAA, GDPR, ISO 27001,
Data sharing	Yes, within AWS	No
Vendor lock-in	AWS Glue is strongly tied to the AWS platform. Usage is billed monthly.	Month to month
Developer tools	Only python and Scala options are available.	REST API, .Net and Python SDKs, PowerShell CLI

Thanks for Reading .Your Suggestions and feedback's are welcome.

AmazonS3Exception: Access Denied errors and potential causes

In last one month have seen this or related S3 access issues couple of times where either unable to upload a file to S3, unable to save S3 inventory report to another bucket or run an Athena query getting data from S3 and every time it took me some time to figure out the cause of the issue.
So, thought of writing this post to make a note of this for future reference.

Whenever we face any S3 access related issues, an obvious reason could be IAM or bucket policies where you don't have the desired access assigned.

Here are my two cents or to troubleshoot this issue:

• First check the AWS IAM permissions, in order to perform any S3 action you should have the required policy defined
• If desired IAM policy is there then check for the target bucket policy
• If the above two are fine then check for the access to S3 encryption key
• If the above three are fine then check for the AWS KMS key policy, as that need to be there for key access

Related reads: 

IAM Policies and Bucket Policies and ACLs!

Overview of KMS Key Policies

In case there are any other scenario, which i'm missing then you're welcome to share and discuss the same in comment section.

That's it... :) 

Amazon EC2 related permission error during AWS RDS database restore

Recently I came across this issue where the database team reported that while trying to restore the Oracle RDS database they are getting an EC2 related permission issue. To my surprise, they also said earlier they had the required permission and this is the first time they came across any such error however I was sure nothing changed from the permissions side.

Just to re-validate, checked for the associated IAM policy and found they have AWS RDSFullAccess policy assigned so logically should be able to restore the RDS database without any issue.

As a second troubleshooting step, I checked CloudTrail logs for any restore failed events without luck then tried to replicate the issue in my test account however didn't encounter any error.

At this point thought to give a deeper look at CloudTrail logs and checked for all the events during the period (when tried to restore the RDS database), interestingly there were a few CreateSecurityGroup related events. 

And when checked further,

Now it was clear that while trying to restore the RDS database, they were selected to create a new security group instead of the desired option "Choose existing Security Group" option.

Post figuring this out, it was easy to make the database team understand that regardless of database vendor the core platform concepts would remain the same.

Some of you might think that why I didn't check CloudTrail logs even before trying to replicate the issue in the test environment and the reason is opinions from others and the name Oracle (now read the above paragraph again 😉).

That's it... 😊

AWS Public IP vs Elastic IP and how can we assign one to EC2 instance

In my previous post I mentioned that in order to make an EC2 instance internet accessible, it should have a Public or Elastic IP assigned. Here anyone new to AWS may wonder that, what is this Elastic IP and How its different from Public IP.

In this post we will discuss about the similarities and differences between these two and how one can assign a Public or Elastic IP to an EC2 instance.

If you are looking from a functional point of view then they both are publicly routable IP addresses and can be used to connect your instance to internet but are different how they persist and the way you can assign one to your instance.

A public IP address is assigned to your instance from Amazon's pool of public IPv4 addresses, and is not associated with your AWS account. When a public IP address is disassociated from your instance, it is released back into the public IPv4 address pool, and you cannot reuse it.

If we put it in simple word then, Public IP addresses are dynamic, which means if you stop/start your instance you get reassigned a new public IP however it would persist if you just reboot the ec2 instance.

Public IP addresses are free and you will not be charged anything for using them.

An Elastic IP address is a static public IPv4 address, designed for dynamic cloud computing. If your instance does not have a public IPv4 address, you can associate an Elastic IP address with your instance to enable communication with the internet.

An Elastic IP address is associated with your AWS account and with it, you can mask the failure of an instance or software by rapidly remapping the address to another instance in your account.

While your instance is running, you are not charged for one Elastic IP address associated with the instance, but you are charged for any additional Elastic IP addresses which is not in use.

Now let’s look at the difference between these two IP types.

1. Elastic IPs are assigned to AWS accounts which you can attach to instances. Public IPs assigned to instances directly.

2. You cannot manually attach or detach public IP from the instance. It’s auto allocated from the pool. Elastic IP can be manually attached and detach from the instance.

3. When an instance is stopped and started again, public IP gets changed. But if the instance is assigned with elastic IP, it will remain the same even if the instance is stopped and started again.

4. If elastic IP is allocated to your account and not in use then you will be charged for it on an hourly basis.

5. Public IP released once your instance is stopped so no question of getting charged for not using it.

6. You won’t be able to re-use same public IP since its allocated from free IP pool. You can always re-use, re-attach elastic IP to other instance when it is released from current instance.

7. You can have maximum 5 elastic IP to your account per region. But, you can have as many public IPs as EC2 instances you spin up.

8. You can have either of them for an instance. If you assign elastic IP to instance then its currently assigned public IP will be released to the free pool.

How would be assign Public or Elastic IP to an instance: 

Public IP: It can be assigned to an instance only during the instance creation time and there are two ways of doing that.

• Edit the Subnet setting and enable Auto-assign Public IP to any EC2 instanced launched in this Subnet.

To do so, From AWS Console => Under Networking & Content Delivery, Select VPC => Now Click on Subnet Tab => Select the intended Subnet and either right Click or from Actions => Select Modify auto-assign IP address

That will open the following Modify auto-assign IP address pop-up, now as shown enable auto-assigning Public IP address.

Now any EC2 instance which would be launched in this subnet would have a public IP assigned.

• You can also assign the Public IP during EC2 instance launch time, you can also alter the default public IP assignment in a subnet from here.

Whatever you select here would over right the default IP assignment settings.

Elastic IP: We need to allocate the Elastic IP address to our AWS account before making use of it.

You can go to Elascic IP windows either from EC2 instance or VPC Dashboard, once you are there then => Select Elastic IP address => Allocate new address

There is not much to discuss here, once you would click on allocate in next screen, It would allocate you an Elastic IP.

Now if you want to assign this IP to any instance, just select it and either click on Actions or Right click on it and Select Allocate address.

That would open the following Associate Address window, from here you can select the intended EC2 instance or specific network interface.

Once select the intended instance, the elastic IP would get associated with the selected instance.

Note: As mentioned in above screenshot, if you associate an Elastic IP address to an EC2 instance which already has a public IP assigned, the public IP is released.

That's it ... :)

Launching your first AWS EC2 instance and making it internet accessible

I assume this is your first EC2 instance and you just have the default VPC with default Security groups, Network Access list and Routing table etc.

As this EC2 is being created in default VPC, which by default has an Internet Gateway attached with a route specified to internet gateway in attached route table so, one only need to complete following steps to make it internet accessible,

• Create this EC2 instance with a Public IP

• If the instance was not created with a public IP then create an Elastic IP and attach it to the EC2 instance

• Identify the attached security group and create a custom inbound rule to allow traffic (RDP and ICMP etc) from an specific IP/Network or from anywhere (0.0.0.0/0)

Here you don’t need to create any outbound rule because Security groups are state full and on the other hand Network access lists by default allow all the traffic.

Note: If you didn’t select a security group during ec2 launch then the ec2 instance would have a new security group name Launch-Wizard-x created and attached, then you need to create custom inbound rules to allow RDP or ICMP traffic for this group.

If required, you can change the attached security group for an instance from here EC2 Dashboard => Network & Security =>Network interfaces.

That's it... :)

Amazon Web Services: A short description of some of the AWS services

As I am planning to sit in my next AWS Certification exam which is Architect Professional in next few months so, thought of starting from scratch and make a note of all common AWS services with their short description (Actually at times I find it hard to recall what service is there for a particular use case so...from next time, I can have a look here).

Amazon Web Services offers a broad set of global cloud-based products including compute, storage, databases, analytics, networking, mobile, developer tools, management tools, IoT, security and enterprise applications. These services help organizations move faster, lower IT costs, and scale.

Compute Services :-

Amazon EC2 : Virtual Servers in the Cloud

Amazon EC2 Auto Scaling : Scale Compute Capacity to Meet Demand

Amazon Container Service : Run and Manage Docker Containers

Amazon Lightsail : Launch and Manage Virtual Private Servers

AWS Batch : Run Batch Jobs at any Scale

AWS Elastic Beanstalk : Run and Manage Web Apps

AWS Lambda : Run your code in Response to events

VMware Cloud on AWS : Build a Hybrid Cloud without Custom Hardware

Storage Services :-

Amazon S3 : Scalable Storage in the Cloud

Amazon EBS : Block Storage for EC2

Amazon EFS : Managed File Storage for EC2

Amazon Glacier : Low cost Archive Storage in Cloud

AWS Storage Gateway : Hybrid Storage Integration

Database Services :-

Amazon RDS :  Managed Relational Database Service For MySQL, PostgreSQL, Oracle, SQL Server and MariaDB

Amazon Aurora : High Performance Relational Database

Amazon DynamoDB : Managed NoSQL Database

Amazon Redshift : Fast, Simple, Cost-effective Data Warehousing

Amazon ElastiCache : In-memory Data Store and Cache

Amazon Neptune :  Fully Managed Graph Database Service

Migration Services :-

AWS Database Migration Service : Migrate Database with Minimal Downtime

AWS Application Discovery Service : Discover On-Premises Application to Streamline Migration

AWS Server Migration Service : Migrate On-Premises Servers to AWS

AWS Snowball : Petabyte-scale Data Transport

AWS Snowball Edge : Petabyte-scale Data Transport with On-board Compute

AWS Snowmobile : Exabyte-scale Data Transport

AWS Migration Hub : Track Migrations from a single place

Networking and Content Delivery Services :-

Amazon VPC : Isolated Cloud Resources

Amazon Route 53 : Scalable Domain Name System

AWS Direct Connect : Dedicated Network Connection to AWS

AWS CloudFront : Global Content Delivery Network

Elastic Load Balancing : High Scale Load Balancing

Amazon API Gateway : Build, Deploy and Manage APIs

Management Tools :-

AWS CloudWatch : Monitor Resources and Application

AWS Auto Scaling : Scale Multiple Resources to Meet Demand

AWS CloudFormation : Create and Manage Resources from Templates

AWS CloudTrail : Track User Activity and API Usage

AWS Config : Track Resources and Inventory

AWS OpsWorks : Automate Operations With Chef and Puppet

AWS Service Catalog : Create and Manage Standardized Products

AWS System Manager : Gain Operational Insight and Track Action

AWS Trusted Advisor : Optimize Performance and Security

AWS Personal Health Dashboard : Personalized View of AWS Service Health

Analytics :-

Amazon EMR : Hosted Hadoop Framework

Amazon Kinesis : Work With Realtime Streaming Data

Amazon Redshift : Simple, Fast, Cost-effective Data Warehousing

Amazon CloudSearch :  Managed Search Service

Amazon Elasticsearch Service :  Run and Scale Elasticsearch Cluster

Amazon Quicksight : Fast Business Analytics Services

AWS Data Pipeline : Orchestration Service for Periodic, Data-driven Workflow

AWS Glue : Prepare and Load Data

Amazon Athena : Query Data in S3 using SQL

Security, Identity & Compliance :-

AWS Identity & Access Management : Manage User Access and Access Keys

AWS Single Sign-On : Aloud SSO Service

AWS Organizations : Policy-based Management for multiple AWS accounts

AWS Key Management Service : Managed Creation and Control Of Encryption Keys

Amazon Inspector : Analyzed Application Security

AWS Shield : DDoS Protection

AWS Guard Duty : Managed Threat Detection Service

AWS WAF : Filter Malicious Web Traffic

......................................................................

...........................................................................

................................................................................

Reference: https://aws.amazon.com/products/

My previous AWS exam related posts can be fond here,

AWS Certified Solution Architect exam experience and few exam related tips

AWS Certified SysOps Administrator - Associate exam experience and few exam related tips

Hope this would be useful for others as well...That's it for now :)

AWS Certified SysOps Administrator - Associate exam experience and few exam related tips

I sat the AWS Certified SysOps Administrator - Associate exam today and passed, in this post I would share my exam experience and a few exam related tips.

There were 55 questions in this exam and and most of the questions were quite trickier than AWS Solution Architect - Associate exam and I hardly found any definitive question however that doesn't mean you would not read and understand definitions of different services ;) 

There were scenario based questions about changing tenancy of a running instance, placement group, EBS snapshots, Instance/ephemeral storage, ELB, Auto-scaling, Route53 records, Routing policy, IAM policies, S3 storage related backup scenario, network connectivity related questions in VPC, Cloudwatch alarms, health checks, Multi-AZ RDS & Read replica, Elastic beanstalk, Cloud Formation, Opswork and AWS shared responsibility model etc.

Here is my exam score, Overall Score: 83%

Topic Level Scoring:
1.0 Monitoring and Metrics: 87%
2.0 High Availability: 100%
3.0 Analysis: 55%
4.0 Deployment and Provisioning: 87%
5.0 Data Management: 85%
6.0 Security: 75%
7.0 Networking: 100%

About the preparation of this exam, learn ins and out of all the AWS services covered in this exam by heart and the best option to do that is, watch this certification's training CBTs few times and once you are done with that now review the FAQ of these services and read AWS security withe paper once. 

I used Ryan Kroonenburg aka AcloudGuru's ( AWS Certified SysOps Administrator - Associate  online ILVT course and Elias Khnaser's AWS Certified SysOps Administrator - Associate course on Pluralsight however i must say any one of these course is suffice to pass the exam.

I would also suggest going for AWS solution architect course/exam first, as it would give you a better understanding about most of the AWS services and provide a solid base for this SysOps exam.

My Previous post about AWS Certified Solution Architect exam experience and few exam related tips can be found here.

Finally before going for the exam, make sure you also practiced on various sample questions available over web, however don't depend on then them as you would find many wrong answers there (however you will be able to to catch them only if you know the ins and out of different serveries by heart).

Hope this would be helpful in passing you AWS Sysops exam....Best of Luck!

That's it.... :)

AWS Certified Solution Architect exam experience and few exam related tips

Last week I attempted and passed my AWS Certified Solution Architect exam, in this post I would share my exam experience and a few exam related tips.

I have never been as anxious and scared as I was before this exam and the obvious reason was not having much hands on experience on AWS platform and the exam feedback related posts which I read on different AWS forums where people are talking about how though these AWS exams are however once the exam started and after looking at first few questions all nervousness instantly went away. Now After passing this exam I can say the questions were very fair neither lengthy nor very hard and what I mean by this is, if you study all the AWS services which are covered in this exam and understand the in and out of them, then you are going to pass this exam easily.

Here is my exam score, Overall Score: 98%

Topic Level Scoring:
1.0 Designing highly available, cost efficient, fault tolerant, scalable systems : 96%
2.0 Implementation/Deployment: 100%
3.0 Security: 100%
4.0 Troubleshooting: 100%

I was preparing for AWS CSA as well as Sysops certification from last three - four months and during my preparation I watched ACloud.guru and Plurasight.com CBT videos two – three times as well as read about different AWS service guides on AWS docs site few times times.   

Services I focused for this particular exam : 

S3	IAM	EC2	Storage Gateways	VPC
Route53	CloudWatch	CloudTrail	CloudFront	ELB
CloudFormation	AutoScalling	ElasticBeanstalk	RDS	DynamoDB
SQS	SNS	SES	Redshift

This exam had 60 questions and out of which around 30 questions were scenario based, where you would be tested about your knowledge of different AWS services…Which service fits best where.  

Beside above listed services, read and understand following one page docs, you would find scenario based questions related to these topics  

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-creating-snapshot.html

http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.html

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html

https://aws.amazon.com/premiumsupport/knowledge-center/snapshot-ebs-raid-array/

http://docs.aws.amazon.com/autoscaling/latest/userguide/as-instance-termination.html

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Security.html

https://d0.awsstatic.com/whitepapers/aws-security-best-practices.pdf ,   Page 2

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-nat-comparison.html

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Internet_Gateway.html

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-nat-gateway.html

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_NAT_Instance.html

http://cloudacademy.com/blog/aws-bastion-host-nat-instances-vpc-peering-security/

https://aws.amazon.com/premiumsupport/knowledge-center/instance-store-vs-ebs/

http://docs.aws.amazon.com/emr/latest/DeveloperGuide/emr-kinesis.html

https://aws.amazon.com/directconnect/   

And most importantly don’t forget to read the FAQ of S3, VPC, EC2, Route53 and RDS/DynmoDB as well as Security related whitepaper.

I also found this post on A cloud guru very useful, https://acloud.guru/forums/aws-certified-solutions-architect-associate/discussion/-KSDNs4nfg5ikp6yBN9l/exam_feedback_-_20_specific_po

Hope this would be helpful in passing your AWS CSA exam....Best of Luck!

That's it.... :)