AWS CloudWatch offers centralized logging, monitoring, and analysis to make the developer’s job easier. A question that comes up for the Enterprise Applications that follow a hybrid cloud deployment (that is, one or more of the application components reside on-premise) is how can we use CloudWatch logs for the on-premise components? Is it even possible? The short answer is yes. And, we will see in this post how to do that.

Why Use CloudWatch For On-Premise Components?

There are several benefits of using CloudWatch for On-Premise components.

• Leverage from centralized logging: You can use the same capabilities for storing the logs centrally for the on-premise components that you are using for the rest of your Cloud-based components.
• Time Conversion: CloudWatch logs are stored in UTC. So, you do not have to worry about tedious conversions, which could often take up your precious time when analyzing logs across several components, especially if these are spread across geographically.
• Consistent Analysis: You can use the same tools and techniques that you are using for Cloud-based components.
• Avoid issues like logs rollover or difficult to access logs. Often, on-premise components are managed by customers or other clients that may require some coordination and effort.

How To Publish To CloudWatch Logs From On-Premise Components?

There are a couple of approaches to accomplish this.

1. Using the AWS CloudWatch Agent to publish logs: This can be extremely useful for on-premise components that follow an appliance model for deployment (such as a pre-baked image with the application components and dependencies). So, this approach is more configuration-centric and should not require code-level changes. Apart from log collection, the CloudWatch agent can also help in capturing system metrics (such as CPU and memory utilization).
2. Using the CloudWatch Logs API to publish logs: This approach requires enhancing the code to use the CloudWatch Logs API to publish logs. Of course, you can make it a reusable module or consider using a third-party library. But, the point is this is a code-centric approach that offers more flexibility.

In this post, we will be talking about using the CloudWatch Logs API. If you are interested in using the CloudWatch Agent, please refer to the AWS CloudWatch Agent documentation for details.

Using The CloudWatch Logs API To Publish Logs

The following code shows the CloudWatch Logs API usage.

package com.cloudnineapps.samples.aws;

import java.util.ArrayList;
import java.util.List;

import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
import com.amazonaws.services.logs.AWSLogsClient;
import com.amazonaws.services.logs.AWSLogsClientBuilder;
import com.amazonaws.services.logs.model.CreateLogGroupRequest;
import com.amazonaws.services.logs.model.CreateLogStreamRequest;
import com.amazonaws.services.logs.model.DescribeLogGroupsRequest;
import com.amazonaws.services.logs.model.DescribeLogGroupsResult;
import com.amazonaws.services.logs.model.DescribeLogStreamsRequest;
import com.amazonaws.services.logs.model.DescribeLogStreamsResult;
import com.amazonaws.services.logs.model.InputLogEvent;
import com.amazonaws.services.logs.model.PutLogEventsRequest;
import com.amazonaws.services.logs.model.PutRetentionPolicyRequest;

/**
 * Sample client for AWS CloudWatch Logs API.
 */
public class AWSCloudWatchLogsSampleClient {

	/** The log group name. */
	private static final String LOG_GROUP = "/myapp/onprem/component-1";

	/** The log stream name. */
	private static final String LOG_STREAM = "app-log";
	
	/** The log retention period (in days). */
	private static final int LOG_RETENTION_PERIOD = 1;

	/** The AWS region. */
	private static String Region = "us-east-1";
	
	/** The CloudWatch client. */
	private static AWSLogsClient Client;
	
	
	/** Opens the CloudWatch log. */
	public static void openCloudWatchLog() throws Exception {
		AWSCredentialsProvider creds = new DefaultAWSCredentialsProviderChain();
		Client = (AWSLogsClient) AWSLogsClientBuilder.standard()
				     .withCredentials(creds)
				     .withRegion(Region)
				     .build();
		// Create and set up the log group if it doesn't exist
		DescribeLogGroupsRequest request = new DescribeLogGroupsRequest().withLogGroupNamePrefix(LOG_GROUP);
		DescribeLogGroupsResult result = Client.describeLogGroups(request);
		if (result.getLogGroups().isEmpty()) {
			CreateLogGroupRequest logGroupRequest = new CreateLogGroupRequest(LOG_GROUP);
			Client.createLogGroup(logGroupRequest);
			PutRetentionPolicyRequest policyRequest = new PutRetentionPolicyRequest(LOG_GROUP, LOG_RETENTION_PERIOD);
			Client.putRetentionPolicy(policyRequest);
			CreateLogStreamRequest logStreamRequest = new CreateLogStreamRequest(LOG_GROUP, LOG_STREAM);
			Client.createLogStream(logStreamRequest);
			log("Created the log group and the log stream.");
		}
	}
	
	/** Logs the specified message. */
	public static void log(String msg) throws Exception {
		// Retrieve the sequence token in the log stream
		DescribeLogStreamsRequest request = new DescribeLogStreamsRequest().withLogGroupName(LOG_GROUP).withLogStreamNamePrefix(LOG_STREAM);
		DescribeLogStreamsResult result = Client.describeLogStreams(request);
		String seqToken = result.getLogStreams().get(0).getUploadSequenceToken();

		// Write to the log stream
		List<InputLogEvent> logEvents = new ArrayList<InputLogEvent>();
		InputLogEvent logEvent = new InputLogEvent().withMessage(msg).withTimestamp(System.currentTimeMillis());
		logEvents.add(logEvent);
		PutLogEventsRequest logRequest = new PutLogEventsRequest(LOG_GROUP, LOG_STREAM, logEvents).withSequenceToken(seqToken);
		Client.putLogEvents(logRequest);
	}
		
	/** Main */
	public static void main(String[] args) throws Exception {
		System.out.println("Launching the application...");
		openCloudWatchLog();
		// Sample log statements
		log("Starting the app...");
		log("Another message");
		System.out.println("Execution completed.");
	}
}

Let’s walk through the code. You can check out the Resources section for the complete code (including the maven pom that can be used to compile and execute).

• The main() method invokes the openCloudWatchLog() method to initialize the CloudWatch Logs SDK client.
• The openCloudWatchLog() method checks if the required Log Group (/myapp/onprem/component-1) exists using the DescribeLogGroupsRequest and the Client.describeLogGroups() call. If not, it creates the Log Group using the CreateLogGroupRequest and the Client.createLogGroup() call. We should always ensure that an appropriate log retention period is set on the log group to avoid accruing a huge log that can lead to a high cost. This is accomplished using the PutRetentionPolicyRequest and the Client.putRetentionPolicy() call. Then, we create the Log Stream using the CreateLogStreamRequest and the Client.createLogStream() call. The following screenshot shows the Log Group in the CloudWatch Console.
  
• Here is a screenshot of the Log Stream under the Log Group.
  
• Next, the code uses the log() method to log sample messages. It uses DescribeLogStreamsRequest and the Client.describeLogStreams() call to retrieve the Log Stream and fetch the upload sequence token. This token must be included when publishing logs (except for the very first publish). Then, we are creating an InputLogEvent with the supplied message and timestamp. The log is published using the PutLogEventsRequest and the Client.putLogEvents() call. As you might have noticed, you do not have to publish individual log statements. You could very well add multiple InputLogEvent objects to publish a batch of logs. The following screenshot demonstrates a sample run of the code.
  

Using IAM Policy To Restrict Access To Specific Log

As part of such a setup, it is important to use restrictive access so that the on-premise component can only access the specific log. The good thing is you can enforce this using IAM as follows.

• Create one or more IAM users for the on-premise components. Grant these users programmatic access only.
• Create a custom policy (or assign an inline policy) like the one shown below and assign it to the above IAM user(s).

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "CloudWatchLogGroupQueryAccess",
            "Effect": "Allow",
            "Action": [
                "logs:DescribeLogGroups"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        },
        {
            "Sid": "CloudWatchLogsAccess",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:PutRetentionPolicy",
                "logs:DescribeLogStreams",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:log-group:/myapp/onprem/*:log-stream:*"
        }
    ]
}

This policy grants access to the required CloudWatch Logs API calls on the on-premise logs only.

Best Practices For Publishing Logs From On-Premise Components

The following are some key best practices to consider.

• Logging can easily get very network intensive. Hence, be quite judicious about which logs are sent to CloudWatch logs. For example, ERROR and WARNING logs are good candidates, but DEBUG is not typically.
• Avoid logging any sensitive data. This is critical, and often not as well thought. For example, some of the things to avoid are logging passwords in plain text, users’ Personally Identifiable Information (PII), and so on.
• Always set an appropriate log retention period on the Log Group.
• Use a well-defined naming convention for the Log Group and Log Streams. For example, /myapp/onprem/component-1.
• Use a restrictive IAM policy for the user that is used to publish logs, and ensure it has access to the component-specific logs only.
• Prefer using application-specific IAM user(s) for logging across multiple applications. This way, you can track and manage access better.

Conclusion

When designing hybrid cloud or on-premise components, evaluate publishing logs to CloudWatch. By following a few key best practices to ensure this is done in a manner that meets the application needs as well as enterprise readiness considerations like security and performance, this can be quite helpful in pro-active application monitoring and management.

Resources

• AWSCloudWatchLogsSampleClient GitHub Repository

Happy logging!
– Nitin

If you liked this post, you will find my AWS Advanced For Developers course helpful that focuses on many such best practices and techniques to design and deploy real-world applications in AWS.

The post How To Use AWS CloudWatch With On-Premise Application Components? appeared first on Cloud Nine Apps.

AWS CloudFormation (CFN) makes it easy to deploy and manage your application infrastructure as an atomic unit using CloudFormation templates. In this article, we will cover how to use CFN to create a multi-tier stack. We will also see how to handle different deployment variations, such as a full-blown production stack with a load balancer, and a smaller footprint development stack using the same CFN template! Lastly, I will also highlight some important tips when designing the CFN templates for your applications.

Note: If you are new to CloudFormation, I highly recommend reading the AWS CloudFormation – An Architect’s Best Friend first to familiarize yourself with the basics.

Identify The Application Deployment Models

Before you start designing the CFN template, it is important to understand in which all possible ways the application can be deployed. At a minimum, identify the key deployment models. For example, what would a typical development deployment look like? Which resources would be needed and what are their configurations? Likewise, for production. Having these details upfront has several benefits.

• You will have a clear understanding of the legitimate combinations in which the application can be deployed.
• You can then delve into the resource configurations and addressing other key aspects like security.
• Lastly, you can ensure that the deployment models are as cost-optimal as possible. For example, the development stack may have the smallest footprint possible to keep cost low versus the production stack that may have a larger footprint.

Let’s take a look at our sample multi-tier stack.

• For the sake of discussion, we will cover 2 deployment models: Development and Production.
• The Production deployment will comprise of a public-facing load balancer, which will be backed by 2 web-tier EC2 instances running Apache. These instances will talk to the EC2 instance hosting the app-tier running tomcat.
• Security Groups will be used for access control. The PublicWebSecurityGroup will be assigned to the web-tier and the AppSecurityGroup will be assigned to the app-tier. Now, you may have noticed the PrivateWebSecurityGroup. Can you guess what’s that for? Perhaps you got it. It is the Security Group created for the Production stack to ensure that the web-tier EC2 instances are only accessible via the load balancer and not exposed publicly.
• The Development deployment of the stack will comprise of a single web-tier EC2 instance and a single app-tier EC2 instance.

So, as you can see, this seemingly simple stack deployment can also become complex considering the different deployment models. But, not to worry. CFN provides us several useful capabilities to make this work.

The Multi-Tier Stack CFN Template

Here is the CFN template.

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "A multi-tier stack instance.",
  "Parameters": {
    "DeploymentType": {
      "Description": "The deployment type.",
      "Type": "String",
      "AllowedValues": ["Development", "Production"],
      "Default": "Development"
    },
    "VPC": {
      "Description": "The VPC for the EC2 instances.",
      "Type": "AWS::EC2::VPC::Id"
    },
    "Subnet1": {
      "Description": "The subnet1 for the EC2 instances.",
      "Type": "AWS::EC2::Subnet::Id"
    },
    "Subnet2": {
      "Description": "The subnet2 for the EC2 instances.",
      "Type": "AWS::EC2::Subnet::Id"
    },
    "SSHSecurityGroup": {
      "Description": "The SSH Security Group for the EC2 instances.",
      "Type": "AWS::EC2::SecurityGroup::Id"
    },
    "KeyPair": {
      "Description": "The key pair name to use to connect to the EC2 instances.",
      "Type": "String"
    }
  },
  "Mappings": {
    "Globals": {
      "Constants": {
        "ImageId": "ami-0b898040803850657",
        "AssignPublicIP": "true",
        "WebInstanceSuffix": "web",
        "AppInstanceSuffix": "app"
      }
    },
    "DeploymentTypes": {
      "Development": {
        "InstanceType": "t2.small",
        "StorageSize": "20"
      },
      "Production": {
        "InstanceType": "t2.medium",
        "StorageSize": "50"
      }
    }
  },
  "Conditions": {
    "CreateMultipleInstances": {"Fn::Not": [{"Fn::Equals": ["Development", {"Ref": "DeploymentType"}]}]}
  },
  "Resources": {
    "LoadBalancer": {
      "Type": "AWS::ElasticLoadBalancing::LoadBalancer",
      "Condition": "CreateMultipleInstances",
      "Properties": {
        "Instances": [{"Ref": "Web1EC2Instance"}, {"Ref": "Web2EC2Instance"}],
        "Subnets": [{"Ref": "Subnet1"}, {"Ref": "Subnet2"}],
        "SecurityGroups": [{"Ref": "PublicWebSecurityGroup"}],
        "Listeners": [{
          "LoadBalancerPort": 80,
          "InstancePort": 80,
          "Protocol": "HTTP"
        }],
        "HealthCheck": {
          "Target": "HTTP:80/",
          "HealthyThreshold": "3",
          "UnhealthyThreshold": "5",
          "Interval": "10",
          "Timeout": "3"
        }
      },
      "Metadata": {
        "AWS::CloudFormation::Designer": {
          "id": "fbf8f065-5dc7-4850-bb4f-8c4287a8cb7b"
        }
      }
    },
    "Web1EC2Instance": {
      "Type": "AWS::EC2::Instance",
      "Properties": {
        "ImageId": {"Fn::FindInMap": ["Globals", "Constants", "ImageId"]},
        "InstanceType": {"Fn::FindInMap": ["DeploymentTypes", {"Ref": "DeploymentType"}, "InstanceType"]},
        "NetworkInterfaces": [{
          "DeviceIndex": "0",
          "SubnetId": {"Ref": "Subnet1"},
          "AssociatePublicIpAddress": {"Fn::FindInMap": ["Globals", "Constants", "AssignPublicIP"]},
          "GroupSet": [{"Ref": "SSHSecurityGroup"}, {"Fn::If": ["CreateMultipleInstances", {"Ref": "PrivateWebSecurityGroup"}, {"Ref": "PublicWebSecurityGroup"}]}]
        }],
        "BlockDeviceMappings": [{
          "DeviceName": "/dev/sdm",
          "Ebs": {
            "VolumeType": "gp2",
            "VolumeSize": {"Fn::FindInMap": ["DeploymentTypes", {"Ref": "DeploymentType"}, "StorageSize"]},
            "DeleteOnTermination": "true"
          }
        }],
        "KeyName": {"Ref": "KeyPair"},
        "Tags": [{"Key": "Name", "Value": {"Fn::Join": ["-", [{"Ref": "AWS::StackName"}, {"Fn::FindInMap": ["Globals", "Constants", "WebInstanceSuffix"]}, "1"]]}}],
        "UserData": {"Fn::Base64": {"Fn::Join": ["", [
          "#!/bin/bash\n",
          "yum install -y aws-cfn-bootstrap\n",
          "\n",
          "# Install the software\n",
          "/opt/aws/bin/cfn-init -v",
          " --stack ", {"Ref": "AWS::StackName"},
          " --resource Web1EC2Instance",
          " --configsets Install",
          " --region ", {"Ref": "AWS::Region"}, "\n",
          "\n",
          "# Signal resource creation completion\n",
          "/opt/aws/bin/cfn-signal -e $?",
          " --stack ", {"Ref": "AWS::StackName"},
          " --resource Web1EC2Instance",
          " --region ", {"Ref": "AWS::Region"}, "\n"
        ]]}}
      },
      "CreationPolicy": {
        "ResourceSignal": {
          "Count": 1,
          "Timeout": "PT5M"
        }
      },
      "DependsOn": "AppEC2Instance",
      "Metadata": {
        "AWS::CloudFormation::Designer": {
          "id": "83fb66e0-bfdf-4076-9d59-3f1077f47a2a"
        },
        "AWS::CloudFormation::Init": {
          "configSets": {
            "Install": ["Install"]
          },
          "Install": {
            "packages": {
              "yum": {
                "httpd": []
              }
            },
            "files": {
              "/var/www/html/index.html": {
                "content": {"Fn::Join": ["", [
                  "<html>\n",
                  "  <head>\n",
                  "    <title>Welcome to a sample multi-tier app!</title>\n",
                  "  </head>\n",
                  "  <body>\n",
                  "    <h1>Welcome to a sample multi-tier app!</h1>\n",
                  "  </body>\n",
                  "</html>\n"
                ]]},
                "mode": "0600",
                "owner": "apache",
                "group": "apache"
              }
            },
            "services": {
              "sysvinit": {
                "httpd": {"enabled": "true", "ensureRunning": "true"}
              }
            }
          }
        }
      }
    },
    "Web2EC2Instance": {
      "Type": "AWS::EC2::Instance",
      "Condition": "CreateMultipleInstances",
      "Properties": {
        "ImageId": {"Fn::FindInMap": ["Globals", "Constants", "ImageId"]},
        "InstanceType": {"Fn::FindInMap": ["DeploymentTypes", {"Ref": "DeploymentType"}, "InstanceType"]},
        "NetworkInterfaces": [{
          "DeviceIndex": "0",
          "SubnetId": {"Ref": "Subnet2"},
          "AssociatePublicIpAddress": {"Fn::FindInMap": ["Globals", "Constants", "AssignPublicIP"]},
          "GroupSet": [{"Ref": "SSHSecurityGroup"}, {"Ref": "PrivateWebSecurityGroup"}]
        }],
        "BlockDeviceMappings": [{
          "DeviceName": "/dev/sdm",
          "Ebs": {
            "VolumeType": "gp2",
            "VolumeSize": {"Fn::FindInMap": ["DeploymentTypes", {"Ref": "DeploymentType"}, "StorageSize"]},
            "DeleteOnTermination": "true"
          }
        }],
        "KeyName": {"Ref": "KeyPair"},
        "Tags": [{"Key": "Name", "Value": {"Fn::Join": ["-", [{"Ref": "AWS::StackName"}, {"Fn::FindInMap": ["Globals", "Constants", "WebInstanceSuffix"]}, "2"]]}}],
        "UserData": {"Fn::Base64": {"Fn::Join": ["", [
          "#!/bin/bash\n",
          "yum install -y aws-cfn-bootstrap\n",
          "\n",
          "# Install the software\n",
          "/opt/aws/bin/cfn-init -v",
          " --stack ", {"Ref": "AWS::StackName"},
          " --resource Web2EC2Instance",
          " --configsets Install",
          " --region ", {"Ref": "AWS::Region"}, "\n",
          "\n",
          "# Signal resource creation completion\n",
          "/opt/aws/bin/cfn-signal -e $?",
          " --stack ", {"Ref": "AWS::StackName"},
          " --resource Web2EC2Instance",
          " --region ", {"Ref": "AWS::Region"}, "\n"
        ]]}}
      },
      "CreationPolicy": {
        "ResourceSignal": {
          "Count": 1,
          "Timeout": "PT5M"
        }
      },
      "DependsOn": "AppEC2Instance",
      "Metadata": {
        "AWS::CloudFormation::Designer": {
          "id": "4e1f2401-833d-442d-be04-89fac2d74778"
        },
        "AWS::CloudFormation::Init": {
          "configSets": {
            "Install": ["Install"]
          },
          "Install": {
            "packages": {
              "yum": {
                "httpd": []
              }
            },
            "files": {
              "/var/www/html/index.html": {
                "content": {"Fn::Join": ["", [
                  "<html>\n",
                  "  <head>\n",
                  "    <title>Welcome to a sample multi-tier app!</title>\n",
                  "  </head>\n",
                  "  <body>\n",
                  "    <h1>Welcome to a sample multi-tier app!</h1>\n",
                  "  </body>\n",
                  "</html>\n"
                ]]},
                "mode": "0600",
                "owner": "apache",
                "group": "apache"
              }
            },
            "services": {
              "sysvinit": {
                "httpd": {"enabled": "true", "ensureRunning": "true"}
              }
            }
          }
        }
      }
    },
    "AppEC2Instance": {
      "Type": "AWS::EC2::Instance",
      "Properties": {
        "ImageId": {"Fn::FindInMap": ["Globals", "Constants", "ImageId"]},
        "InstanceType": {"Fn::FindInMap": ["DeploymentTypes", {"Ref": "DeploymentType"}, "InstanceType"]},
        "NetworkInterfaces": [{
          "DeviceIndex": "0",
          "SubnetId": {"Ref": "Subnet1"},
          "AssociatePublicIpAddress": {"Fn::FindInMap": ["Globals", "Constants", "AssignPublicIP"]},
          "GroupSet": [{"Ref": "SSHSecurityGroup"}, {"Ref": "AppSecurityGroup"}]
        }],
        "BlockDeviceMappings": [{
          "DeviceName": "/dev/sdm",
          "Ebs": {
            "VolumeType": "gp2",
            "VolumeSize": {"Fn::FindInMap": ["DeploymentTypes", {"Ref": "DeploymentType"}, "StorageSize"]},
            "DeleteOnTermination": "true"
          }
        }],
        "KeyName": {"Ref": "KeyPair"},
        "Tags": [{"Key": "Name", "Value": {"Fn::Join": ["-", [{"Ref": "AWS::StackName"}, {"Fn::FindInMap": ["Globals", "Constants", "AppInstanceSuffix"]}, "1"]]}}],
        "UserData": {"Fn::Base64": {"Fn::Join": ["", [
          "#!/bin/bash\n",
          "yum install -y aws-cfn-bootstrap\n",
          "\n",
          "# Install the software\n",
          "/opt/aws/bin/cfn-init -v",
          " --stack ", {"Ref": "AWS::StackName"},
          " --resource AppEC2Instance",
          " --configsets Install",
          " --region ", {"Ref": "AWS::Region"}, "\n",
          "\n",
          "# Signal resource creation completion\n",
          "/opt/aws/bin/cfn-signal -e $?",
          " --stack ", {"Ref": "AWS::StackName"},
          " --resource AppEC2Instance",
          " --region ", {"Ref": "AWS::Region"}, "\n"
        ]]}}
      },
      "CreationPolicy": {
        "ResourceSignal": {
          "Count": 1,
          "Timeout": "PT5M"
        }
      },
      "Metadata": {
        "AWS::CloudFormation::Designer": {
          "id": "4720705c-1add-4c63-abd6-d9fd4626a43d"
        },
        "AWS::CloudFormation::Init": {
          "configSets": {
            "Install": ["Install"]
          },
          "Install": {
            "packages": {
              "yum": {
                "tomcat": [],
                "tomcat-webapps": []
              }
            },
            "services": {
              "sysvinit": {
                "tomcat": {"enabled": "true", "ensureRunning": "true"}
              }
            }
          }
        }
      }
    },
    "PublicWebSecurityGroup": {
      "Type": "AWS::EC2::SecurityGroup",
      "Properties": {
        "GroupName": {"Fn::Join": ["-", [{"Ref": "AWS::StackName"}, "public-web-sg"]]},
        "GroupDescription": {"Fn::Join": ["", ["Enables public web access for ", {"Ref": "AWS::StackName"}, "."]]},
        "VpcId": {"Ref": "VPC"},
        "SecurityGroupIngress": [
          {"IpProtocol": "tcp", "FromPort": "80", "ToPort": "80", "CidrIp": "0.0.0.0/0"}
        ],
        "Tags": [{"Key": "Name", "Value": {"Fn::Join": ["-", [{"Ref": "AWS::StackName"}, "public-web-sg"]]}}]
      },
      "Metadata": {
        "AWS::CloudFormation::Designer": {
          "id": "bfef096b-3b53-4799-b880-0df21011e7ed"
        }
      }
    },
    "PrivateWebSecurityGroup": {
      "Type": "AWS::EC2::SecurityGroup",
      "Properties": {
        "GroupName": {"Fn::Join": ["-", [{"Ref": "AWS::StackName"}, "private-web-sg"]]},
        "GroupDescription": {"Fn::Join": ["", ["Enables private web access for ", {"Ref": "AWS::StackName"}, "."]]},
        "VpcId": {"Ref": "VPC"},
        "SecurityGroupIngress": [
          {"IpProtocol": "tcp", "FromPort": "80", "ToPort": "80", "SourceSecurityGroupId": {"Ref": "PublicWebSecurityGroup"}}
        ],
        "Tags": [{"Key": "Name", "Value": {"Fn::Join": ["-", [{"Ref": "AWS::StackName"}, "private-web-sg"]]}}]
      },
      "Metadata": {
        "AWS::CloudFormation::Designer": {
          "id": "f00bae41-3fe2-41c1-ad14-64680744f71f"
        }
      }
    },
    "AppSecurityGroup": {
      "Type": "AWS::EC2::SecurityGroup",
      "Properties": {
        "GroupName": {"Fn::Join": ["-", [{"Ref": "AWS::StackName"}, "app-sg"]]},
        "GroupDescription": {"Fn::Join": ["", ["Enables access to ", {"Ref": "AWS::StackName"}, " app tier."]]},
        "VpcId": {"Ref": "VPC"},
        "SecurityGroupIngress": [
          {"IpProtocol": "tcp", "FromPort": "8080", "ToPort": "8080", "SourceSecurityGroupId": {"Fn::If": ["CreateMultipleInstances", {"Ref": "PrivateWebSecurityGroup"}, {"Ref": "PublicWebSecurityGroup"}]}}
        ],
        "Tags": [{"Key": "Name", "Value": {"Fn::Join": ["-", [{"Ref": "AWS::StackName"}, "app-sg"]]}}]
      },
      "Metadata": {
        "AWS::CloudFormation::Designer": {
          "id": "adf9b8b7-25c4-4ddb-9401-e5c34da55511"
        }
      }
    }
  },
  "Outputs": {
    "StackURL": {
      "Description": "The stack web URL.",
      "Value": {"Fn::Join": ["", ["http://",
          {"Fn::If": ["CreateMultipleInstances", {"Fn::GetAtt": ["LoadBalancer", "DNSName"]}, {"Fn::GetAtt": ["Web1EC2Instance", "PublicIp"]}]}
      ]]}
    }
  },
  "Metadata": {
    "AWS::CloudFormation::Designer": {
      "fbf8f065-5dc7-4850-bb4f-8c4287a8cb7b": {
        "size": {
          "width": 60,
          "height": 60
        },
        "position": {
          "x": 560,
          "y": 50
        },
        "z": 0,
        "embeds": [],
        "isassociatedwith": [
          "83fb66e0-bfdf-4076-9d59-3f1077f47a2a",
          "4e1f2401-833d-442d-be04-89fac2d74778",
          "bfef096b-3b53-4799-b880-0df21011e7ed"
        ]
      },
      "83fb66e0-bfdf-4076-9d59-3f1077f47a2a": {
        "size": {
          "width": 60,
          "height": 60
        },
        "position": {
          "x": 470,
          "y": 150
        },
        "z": 0,
        "embeds": []
      },
      "4e1f2401-833d-442d-be04-89fac2d74778": {
        "size": {
          "width": 60,
          "height": 60
        },
        "position": {
          "x": 650,
          "y": 150
        },
        "z": 0,
        "embeds": []
      },
      "bfef096b-3b53-4799-b880-0df21011e7ed": {
        "size": {
          "width": 60,
          "height": 60
        },
        "position": {
          "x": 460,
          "y": 50
        },
        "z": 0,
        "embeds": []
      },
      "c4db756b-b07b-4324-be5f-6ed116047ed8": {
        "source": {
          "id": "fbf8f065-5dc7-4850-bb4f-8c4287a8cb7b"
        },
        "target": {
          "id": "bfef096b-3b53-4799-b880-0df21011e7ed"
        },
        "z": 11
      },
      "4720705c-1add-4c63-abd6-d9fd4626a43d": {
        "size": {
          "width": 60,
          "height": 60
        },
        "position": {
          "x": 553,
          "y": 265
        },
        "z": 0,
        "embeds": []
      },
      "a60d3d35-5a1d-4a58-a2bd-6f5ffe94017a": {
        "source": {
          "id": "83fb66e0-bfdf-4076-9d59-3f1077f47a2a"
        },
        "target": {
          "id": "4720705c-1add-4c63-abd6-d9fd4626a43d"
        },
        "z": 11
      },
      "f00bae41-3fe2-41c1-ad14-64680744f71f": {
        "size": {
          "width": 60,
          "height": 60
        },
        "position": {
          "x": 660,
          "y": 270
        },
        "z": 1,
        "embeds": []
      },
      "adf9b8b7-25c4-4ddb-9401-e5c34da55511": {
        "size": {
          "width": 60,
          "height": 60
        },
        "position": {
          "x": 540,
          "y": 360
        },
        "z": 1,
        "embeds": []
      },
      "5f28f4f3-edee-437e-856b-dda339912a82": {
        "source": {
          "id": "4720705c-1add-4c63-abd6-d9fd4626a43d"
        },
        "target": {
          "id": "adf9b8b7-25c4-4ddb-9401-e5c34da55511"
        },
        "z": 11
      }
    }
  }
}

Let’s break it down by section to understand it better.

Template Parameters

• The DeploymentType parameter is used to determine the deployment model. Such logical parameters are extremely useful as compared to exposing the individual resource properties as parameters. Why? Firstly, these enable the CFN template designer to ensure that the deployments meet only the prescribed models. Secondly, these hide unnecessary complexities from the template users. Also, tomorrow if you add more resources or change configurations, the template users do not have to be bothered about these.
• The next 3 parameters take the target VPC and subnet information. Note that because we are doing a Production deployment we do want to leverage from the high availability capabilities offered by AWS so that we can distribute our EC2 instances across multiple Availability Zones.
• The SSHSecurityGroup parameter is used to specify the Security Group that will be used to connect to the EC2 instances. Now, you could very well create a Security Group for SSH in this template itself. However, I wanted to demonstrate the use of a Security Group that has been created by the networking team for access control. For example, they may set up the SSH Security Group to only allow inbound traffic from the corporate network. So, as opposed to every application team coming up with their own SSH Security Group, here the organization has chosen to use the same SSH Security Group for consistency and ease of management. These considerations are quite important when you design CFN templates. You want to ensure you are not creating backdoors that could lead to a potential security breach.
• The KeyPair parameter specifies the SSH key pair name that will be used for connecting to the EC2 instances.

Mappings

• The Globals map defines constants to avoid hardcoding values all over the template. For example, instead of hardcoding the AMI ID everywhere, we can simply define it once here. Tomorrow, if we want to change it, we just have to update it in one place. Likewise, the other constants.
• The DeploymentTypes map specifies the resource properties for each deployment model. For example, a Development EC2 instance will use t2.small instance type. Whereas, a Production EC2 instance will use t2.medium.

Conditions

• The CreateMultipleInstances condition is set to true only if it is NOT a Development deployment. This will be used later in the template when creating the resources. If you are wondering why I didn’t call it IsProductionDeployment, there is a good reason. What if I add another deployment model tomorrow called QA, which also has multiple instances? By keeping the condition name more generic, we can use it for both these deployment models.

Resources

Notice the use of DependsOn to ensure the resources are created in the correct order.

• First, it creates a load balancer (LB) for the web-tier if the CreateMultipleInstances condition is true. This LB will have 2 EC2 instances attached – Web1EC2Instance and Web2EC2Instance. It will have access to the specified subnets and will be assigned the PublicWebSecurityGroup. The LB will listen on port 80 (HTTP) and will forward traffic to port 80 on the EC2 instances. As we know, LB works by checking the health of the underlying instances. In this case, it will access the base URL (/) every 10 seconds with a timeout value of 3 seconds. If the health check fails 5 consecutive times, the instance will be declared unhealthy. And, the check must succeed 3 consecutive times to declare the instance as healthy.
• Next, it creates the Web1EC2Instance. It uses the Globals map to find some property values like the AMI ID. The InstanceType is set based on the DeploymentType. If it is a Production EC2 instance, it will be assigned the PrivateWebSecurityGroup to avoid direct public access. For Development, it will be assigned the PublicWebSecurityGroup. Also, the SSH Security Group will be assigned. The instance will have a single EBS volume attached. Apart from KeyName and Tags, notice the use of UserData. This is used to install and configure the apache daemon when the instance first boots using the AWS CFN Helper Scripts. The Metadata section contains the information used by these scripts to do their work. The CreationPolicy is used to ensure that CFN waits till it receives a success or failure signal from the UserData script with a timeout of 5 minutes. Note that Web1EC2Instance is created regardless of the DeploymentType since the stack will have at least one web-tier instance.
• The Web2EC2Instance is created only if the CreateMultipleInstances condition is set to true. Its configuration is the same as the Web1EC2Instance.
• The AppEC2Instance is created similarly with the key difference being it will be assigned the AppSecurityGroup and tomcat will be installed on it. Note that just like the web-tier we could create multiple EC2 instances for the app-tier and put an LB in-front of these for high availability and load balancing purposes. But, I’ve skipped that part here for brevity purposes.
• Next, it creates the Security Groups for the respective tiers. Notice how the PrivateWebSecurityGroup sets its ingress rule to permit traffic only from the source resource that has the PublicWebSecurityGroup assigned. This will ensure that only the LB can talk to the web EC2 instances.

Outputs

• The template outputs the stack access URL. Again, it uses the CreateMultipleInstances condition to determine whether the output URL should be based on the load balancer or the Web1EC2Instance.

Metadata

This section contains data used by the CFN Designer tool.

Sample Deployment Commands

The following command shows a Production deployment.

aws cloudformation deploy --profile aws-training --stack-name MyAppProdStack --parameter-overrides DeploymentType=Production VPC=vpc-0ef286bd199748f2a Subnet1=subnet-05eaf962fd4e0e12c Subnet2=subnet-00077ede2ac0521a6 SSHSecurityGroup=sg-0fa7434cec1fd0b2d KeyPair=C9SSHKeyPair --template-file cfns/Multi_Tier_Stack.json

And, here’s a command that shows a Development deployment.

aws cloudformation deploy --profile aws-training --stack-name MyAppDevStack --parameter-overrides DeploymentType=Development VPC=vpc-0ef286bd199748f2a Subnet1=subnet-05eaf962fd4e0e12c Subnet2=subnet-00077ede2ac0521a6 SSHSecurityGroup=sg-0fa7434cec1fd0b2d KeyPair=C9SSHKeyPair --template-file cfns/Multi_Tier_Stack.json

Conclusion

Great job in reading along. As you might have already gathered, authoring a good CFN template is no different from writing good code. It needs to have considerations of what you are trying to accomplish, enterprise readiness aspects like security and be cost-efficient. By thinking about these points upfront you can not only design a better template but also avoid common issues and save time in the longer run.

Happy deploying!
– Nitin

The post How To Create A Multi-Tier Stack Using AWS CloudFormation appeared first on Cloud Nine Apps.

AWS CloudFormation (CFN) helps in automating deployments by delivering Infrastructure as a Code (IaaC). It offers several useful capabilities, like simple templates, support for a wide range of AWS services, dependency management, parallel deployments, and so on. To use CloudFormation, you create CFN templates either from scratch or from samples. You can also use tools like the CloudFormation Designer to author the CFN templates. However, similarly to coding, you will run into errors with CFN as well. In this post, we will cover the two prominent types of CFN errors and how to troubleshoot these.

Types of CloudFormation Errors

There are two types of CFN errors.

1. Syntax Errors: A CFN template is a text document in either JSON or YAML format. The syntax errors in the CFN template lead to this error. For example, a missing comma or brace in JSON, an incorrect indentation in YAML, and so on. Such a template is not even deployed as the CloudFormation Console or the CLI will fail upon detecting the error.
2. Semantic Errors: These occur at deployment time typically due to coding or an infrastructure issue. For example, an incorrectly specified resource property name, trying to create a duplicate resource, and so on.

How To Troubleshoot CloudFormation Syntax Errors?

Let’s take an example here. The following screenshot shows a CFN template snippet in JSON format.

As you can see, the curly brace for the PolicyDocument property is missing. When we try to deploy this CFN template, we will see an error like the one shown below.

Here the JSON parser has detected a missing comma or a curly brace. After you add the missing brace, the error will be resolved. Errors like these are quite common during development. Here are some tips on how to resolve or minimize the occurrence of such errors.

• Check if your text editor supports block matching capabilities, like the powerful VI editor. This can be used to find the matching brace, and if you find that it is not on the expected element you have found the problem area. Then you can keep drilling down till you find the problem. As a general practice, it is also a good idea to always specify the closing brace when opening a brace so that you do not miss it. This can really come in handy for large templates.
• You can also use a JSON validator or beautifier. There are several available online. While in most cases the validator may not reveal any more useful information than what the CFN CLI output shows, the beautifier may be a bit helpful to visually inspect the template for any anomalies.

How To Troubleshoot CloudFormation Semantic Errors?

The semantic errors can range from simple things like incorrect property names, missing resource names to infrastructure-related errors, such as duplicate infrastructure resource name. And, as we discussed earlier, these occur at the deployment time. So, it may increase your end-to-end resolution time. Let’s take the example wherein we are trying to deploy an S3 bucket using a CFN template but a bucket with the specified name already exists. The following screenshot shows the CFN CLI output.

As you can see, the CLI execution failed. However, unlike the syntax error, the details are not available in this output. So, to troubleshoot further, you can either go to the CloudFormation Console or run the command shown in the above output.

aws cloudformation describe-stack-events –stack-name c9apps-demo-bucket1

This will show the stack events in reverse chronological order (most recent event first). Scroll through the list till you find the error cause as shown below.

It shows that a bucket named c9apps-demo-bucket (which was the bucket name we specified when deploying the template) already exists in another stack. This approach can be particularly useful for DevOps pipelines to retrieve the error information.

The other option is to go to the CloudFormation Console and look into the stack events for the error cause as shown below.

As we can see, the console shows the same information.

While semantic errors is a broader category and it is difficult to have a silver bullet solution for every scenario, here are some general tips.

• Use the AWS Resource and Property Types Reference document to check the resources types and properties.
• Use unique resource identifier names in the CFN templates. In general, giving names based on purpose is a good idea. For example, instead of calling it EC2Instance1, a better name would be WebTierInstance1.
• CloudFormation offers several Intrinsic Functions for common computing needs. Leverage from these to make more meaningful and unique resource names. For example, the Fn::Join Intrinsic Function can be used to combine a list of strings.
• Leverage from the lifecycle for choosing the resource names. For example, resources that are global in nature (such as IAM role) may not have the stack name as part of their names. Whereas, resources that are meant for a specific stack only can use the stack name to make their names unique.

Troubleshooting CFN errors takes some practice. Also, at times the error messages may not be easy to understand. So, it is a good idea to familiarize yourself with these techniques so that you are better prepared when the issues occur.

Be a smart troubleshooter!
– Nitin

If you liked this post, you will find my AWS CloudFormation Deep Dive course helpful that focuses on many such best practices, techniques and hands-on examples to use CloudFormation in real-world deployments.

The post How To Troubleshoot AWS CloudFormation Errors appeared first on Cloud Nine Apps.

AWS CloudFormation (CFN) conditions are quite useful for purposes like conditionally create resources, conditionally set the resource properties, and so on. However, when you start getting into some advanced scenarios, these may become a bit limited. For example, checking for the existence of a resource so that you create it only once. How do you handle such scenarios? Is there any way you can do this via CFN as opposed to using some alternatives? That is precisely the purpose of this article.

Note that this is an advanced CFN topic. If you are new to CFN, please check out the AWS CloudFormation – An Architect’s Best Friend to understand the basics first.

What Does Creating A Dynamic Condition Expression Mean?

Simply put, it means creating the expression text dynamically at the deployment time. Typically, the condition expression is static. For example, take a look at the condition below that will evaluate to true if the CreateGlobalResourcesParam parameter is set to “true”.

  "CreateGlobalResources": {"Fn::Equals", ["true", {"Ref": "CreateGlobalResourcesParam"}]}

When creating a Dynamic Condition Expression, the expression text is prepared at the deployment time.

"CreateGlobalResources": <content to prepare the expression text dynamically>

What Is The Need To Create A Dynamic Condition Expression?

To understand the need, let’s take a simple use case. Let’s say you have a CFN template that creates a serverless stack. It comprises a lambda and associated resources. Since the user base is spread across the globe, the plan is to create a stack per target region. For this discussion, let’s use us-east-1 and us-west-1 AWS regions with us-east-1 being the primary region.

So far so good. Now, we know that lambda requires an execution IAM role so that it has access to the required AWS resources. And, there are going to be two lambda deployments (one per region). So, we have two options – either create a lambda role per stack or create a single IAM role (say, in the primary stack) that will be used by all subsequent stacks. Both options have their own merits. The former offers more autonomy to each stack, whereas the second option leverages from the lifecycle of resources and minimizes the sprawl of roles that serve the same purpose. Another advantage of the second option is if there were updates to the role, you only need to update the primary stack and all stacks will automatically get the updates, thus reducing the end-to-end rollout team. Also, there are limits to the number of IAM roles you can create. Although, you may be able to request an increase. This is just one scenario. But, there could be similar other use cases where you are trying to reuse a global resource across stacks.

So, for this discussion let’s say that we do want to create the IAM role only once. Now, let’s talk about how to achieve this by creating a Dynamic Condition Expression.

Using Macro To Create A Dynamic Condition Expression

CFN offers a powerful capability called Macro. If you have ever programmed in a language like C/C++ (or another language that supports pre-processors), you may be already familiar with the use of macros. Essentially, a CFN Macro lets you execute certain logic to modify either the content of the template or a snippet within the template. All macros are evaluated before executing the CFN template. So, these are perfect to inject the condition expression dynamically. The following screenshot demonstrates this.

Let’s understand this better.

• In the Conditions block, we are defining a condition named CreateGlobalResources. It should be set to true if the global resources should be created. Otherwise, it should be set to false.
• It is using the Fn::Transform intrinsic function to invoke the ResourceHelper macro with 2 parameters – Operation and RoleName. We will see the details of the macro later in this post. The Operation parameter specifies the operation to perform (in this case, roleExists, which will check for the existence of the specified role). The RoleName parameter specifies the name of the role to check.
• Here, the macro is expected to return the string “true” only if the role exists. Otherwise, it should return the string “false”. This value is then compared with “false” using the Fn::Equals intrinsic function.
• This condition will be then used later in the template to conditionally create the LambdaFunctionRole role as shown below.
  

The idea here is when the primary stack is deployed in us-east-1, the macro will return “false” as the LambdaFunctionRole role does not exist yet. Hence, the CreateGlobalResources condition will be set to true and the LambdaFunctionRole will be created. When the stack is deployed in us-west-1, the macro will return “true” and the CreateGlobalResources condition will be set to false. Hence, the role will not be created again.

Let’s See The Macro Code

Now that we understand how the Dynamic Condition Expression is created, let’s take a look at the macro code.

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "A stack to provide utility methods for resources.",
  "Parameters": {
    "CreateLambdaRole": {
      "Description": "Whether to create the lambda role.",
      "Type": "String",
      "AllowedValues": ["true", "false"],
      "Default": "true"
    }
  },
  "Conditions": {
    "CreateGlobalResources": {"Fn::Equals": ["true", {"Ref": "CreateLambdaRole"}]}
  },
  "Resources": {
    "ResourceHelper": {
      "Type": "AWS::CloudFormation::Macro",
      "Properties": {
        "Name": {"Ref": "AWS::StackName"},
        "Description": "A macro to provide utility methods for resources.",
        "FunctionName": {"Fn::GetAtt": ["ResourceHelperLambda", "Arn"]}
      }
    },
    "ResourceHelperLambda": {
      "Type": "AWS::Lambda::Function",
      "Properties": {
        "FunctionName": "ResourceHelperLambda",
        "Handler": "index.handler",
        "Runtime": "nodejs8.10",
        "Role": {"Fn::Join": ["", ["arn:aws:iam::", {"Ref": "AWS::AccountId"}, ":role/ResourceHelperLambdaRole"]]},
        "MemorySize": "128",
        "Timeout": "30",
        "Code": {
          "ZipFile": {"Fn::Join": ["", [
            "const AWS = require('aws-sdk');\n",
            "AWS.config.update({region: '", {"Ref": "AWS::Region"}, "'});\n",
            "const iam = new AWS.IAM();\n",
            "\n",
            "exports.handler = (event, context, callback) => {\n",
            "  console.log('handler(): event: ' + JSON.stringify(event));\n",
            "  const params = event.params;\n",
            "  const op = params.Operation;\n",
			" const roleName = params.RoleName;\n",
            "  if (op == 'roleExists') {\n",
            "    var request = {\n",
            "      RoleName: roleName\n",
            "    };\n",
            "    iam.waitFor('roleExists', request, function(err, data) {\n",
            "      if (err) {\n",
            "        // Role does not exist\n",
            "        handleResponse(null, 'true');\n",
            "      }\n",
            "      else if (data.Role) {\n",
            "        // Role exists\n",
            "        handleResponse(null, 'false');\n",
            "      }\n",
            "      else {\n",
            "        handleResponse(new Error(`Could not get information about role '${params.RoleName}'.`), null);\n",
            "      }\n",
            "    });\n",
            "  }\n",
            "  else {\n",
            "    handleResponse(new Error(`Unsupported operation '${Operation}'.`), null);\n",
            "  }\n",
            "  const handleResponse = function(err, data) {\n",
            "    var response = {\n",
            "      status: 'SUCCESS',\n",
            "      requestId: event.requestId,\n",
            "    };\n",
            "    if (err) {\n",
            "      console.log('handleResponse(): ' + err);\n",
            "      response.status = 'FAILURE';\n",
            "    }\n",
            "    else {\n",
            "      response.fragment = data;\n",
            "    }\n",
            "    console.log('handleResponse(): response: ' + JSON.stringify(response));\n",
            "    callback(null, response);\n",
            "  };\n",
            "}\n"
          ]]}
        }
      }
    },
    "ResourceHelperLambdaRole": {
      "Type": "AWS::IAM::Role",
      "Condition": "CreateGlobalResources",
      "Properties": {
        "RoleName": "ResourceHelperLambdaRole",
        "AssumeRolePolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [{
            "Effect": "Allow",
            "Principal": {
              "Service": "lambda.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
          }]
        },
        "ManagedPolicyArns": [
          "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole",
          "arn:aws:iam::aws:policy/IAMReadOnlyAccess"
        ]
      }
    },
    "ResourceHelperLambdaLogGroup": {
      "Type": "AWS::Logs::LogGroup",
      "Properties": {
        "LogGroupName": {"Fn::Join": ["", ["/aws/lambda/", {"Ref": "ResourceHelperLambda"}]]},
        "RetentionInDays": 1
      }
    }
  }
}

As we know, the purpose of this macro is to prepare the Dynamic Condition Expression text with value as either “true” (if the role exists) or “false”. A macro is essentially a lambda-based stack that exists in the same region as the consuming stack. Here, I am using Node.js for implementation. If you are not familiar with it, do not worry. I will explain the important aspects below.

• The macro CFN template takes a single parameter – CreateLambdaRole. It is referred by the CreateGlobalResources condition that is used to conditionally create the IAM role required for the macro lambda. Do not confuse this with the use case we saw earlier. The condition here is strictly for the macro lambda role only. Can you guess why? Perhaps you got it. Since there are 2 AWS regions in our sample deployment, we will have 2 deployments of the macro stack as well. But, we would create the IAM role for the macro lambda only once and use it for both the stacks. So, when the macro stack is created in the primary region (us-east-1), the CreateLambdaRole parameter will be set to “true” and the macro lambda role will be created. And, vice-versa for us-west-1.
• The first resource is the ResourceHelper macro itself.
• The next resource is the ResourceHelperLambda that provides macro implementation.
  • In its Code block, the AWS SDK is initialized first.
  • Next in the handler function, we are first extracting the Operation and RoleName parameters. Apart from these parameters, the macro also receives other request data in a well-defined format, such as the requestId, which is a unique identifier for a given macro call. The following screenshot shows a sample macro request.
    
  • This is followed by a check to confirm if the request operation is roleExists. If so, an IAM SDK call is issued to check whether the supplied role already exists.
  • If the role exists, a string with value ‘true’ is returned. Otherwise, ‘false’ is returned. This string output is then used to prepare the lambda response in a well-defined format, which includes additional fields, such as status. The following screenshot shows a sample macro response. The content of the fragment field is our Dynamic Condition Expression text.
    
  • The ResourceHelperLambdaRole is the IAM role used by the macro lambda.
  • Finally, the ResourceHelperLambdaLogGroup is the log group used by the macro lambda.

In this article, we saw how to use macros to create CFN condition expressions dynamically. This capability can be extremely useful, especially if you are trying to avoid splitting provisioning logic in multiple places, such as some in CFN and some via scripts. If you have multiple such common methods, you could create one or more reusable macros that can be used by multiple stacks thus increasing the reusability. So, next time you have a situation where you would want the CFN conditions to be dynamic, you may want to consider creating Dynamic Condition Expressions using macros.

Happy coding!
– Nitin

If you liked this post, you will find my AWS CloudFormation Deep Dive course useful that focuses on many such best practices, techniques and hands-on examples to use CloudFormation in real-world deployments.

The post How To Create Dynamic Condition Expressions In AWS CloudFormation Using Macros appeared first on Cloud Nine Apps.

Software as a Service (SaaS) has been a predominant model for many software vendors. It helps similarly delivering software as a Cloud vendor delivers infrastructure services. SaaS applications are often deployed on a public cloud, like Amazon Cloud (AWS), Microsoft Azure, Google Cloud and so on. However, at times the organization may choose to use their datacenter (a.k.a. private cloud) to host the SaaS application and leverage their investment in the infrastructure. When you are designing SaaS applications, it takes more than just deploying application bits to the Cloud. Taking proper design considerations can not only help you in accomplishing a good design, but it can also help you in reducing costs and manage your deployment more effectively. In this post, I will cover some key considerations and tips to design SaaS applications that I have found useful over the years.

How Is Designing Applications For Cloud Different From On-premise Applications Design?

That’s a valid question that many Architects face, especially in the early phases. There are certain subtle differences. Let’s look at some of these just to get the perspective.

• Better Modularization: If you have a monolithic application with a huge footprint, it is perhaps wise to see if it can be broken into logical components that can be deployed separately. This not only improves modularity but helps you reduce the footprint of the application. Let’s say you have an application that uses background jobs to refresh data. You could segregate the core application and the background jobs as 2 (or more) components that can be deployed separately. This will reduce the footprint of the core application. So, you could potentially choose a smaller resource size. Also, depending on the needs, you can scale these two independently. So, if there is an increase in demand for background jobs, you can increase its capacity and likewise for the application tier. Since the resource size for each of these is small, scaling only the required tier/component will lead to an overall lower cost than more bulky resources for the monolith. Makes sense?
• Application is always up-to-date: This is a big shift for many on-premise applications. In the Cloud, customers typically expect the application to be always on the latest version. Now, if you think about it as an Architect it means you are not only able to update the application bits but also upgrade customer data with preferably no customer involvement. That is, it is completely transparent to them.

These are just some of the differences. But, you get the point.

Key Design Considerations for SaaS

When you are designing an application for Cloud, you want to consider the following key points.

• Choose appropriate Cloud services: Of course, you are deploying to Cloud. But, which services do you want to use? Are you simply going to use Infrastructure as a Service (IaaS)? Or are you going to take advantage of some Platform as a Service (PaaS) capabilities as well? The answer may not be always straightforward. So, here are some guidelines.
  • Are you going to have the same application deployed to multiple Cloud platforms or on-premise as well? In that case, it may make sense to not have (or minimize) Cloud vendor-specific services and stick to more of IaaS services.
  • Cost should be an important consideration in choosing the services. For example, certain PaaS services that are Cloud vendor managed may be managed by your team itself to reduce cost. While this may not make sense for every service, it is worth exploring.
  • Keep your team expertise in mind. What would it take to use a certain Cloud service? And, if the team were to also maintain the underlying infrastructure, what skills would be needed?
• Design for failure: Designing fault-tolerant and highly available applications is fundamental to Cloud. Assume that your application will run into issues and how you will ensure that it continues to serve the users. These could be application failures or underlying infrastructure failures. There are a few useful capabilities offered by Cloud vendors to help here.
  • Use a Load Balancer: You can put the application nodes behind a Load Balancer for both load-balancing purposes as well as to make sure that even if one or more nodes go down, the application is served via the other nodes.
  • Geographically distributed application: Several Cloud vendors offer capabilities to spread the application across multiple geographical areas so that even if one area is impacted (say, due to a natural disaster), the application can be served from the other areas. For example, AWS supports deploying an application across multiple Availability Zones.
• Modularize your application: As we discussed in the earlier section, segregating components that can be deployed and managed separately can help reduce the footprint of the application and thus reduce the cost of the infrastructure. You may also consider making some of these components as microservices. The microservice approach can be particularly useful if there are other potential consumers besides your application. Now, that does not mean, you go overboard and create unnecessary components. Hence, one way to do this is to make components that can be deployed separately (such as core application versus background jobs).
• Security: Security involves many aspects – from securing your infrastructure to application. Some of the key aspects include ensuring that only the required ports are opened, using as minimal privileges on the resources as possible, having proper role-based access control, use of encryption, and so on. Security should not be looked at as a one-time deal. It is an on-going process that should improve and evolve over time.
• Multi-tenancy: A key benefit of running in the Cloud is being able to serve multiple customers using the same instance of the application. This brings out some obvious challenges in the application design to ensure that each customer’s data is segregated for both security and regulatory purposes. Some teams choose to go with a different persistent storage instance per customer, such as a separate database per customer. And, some choose to segregate data using row-level identifiers. Regardless of which approach you take, it is important to ensure that the architecture meets the scalability and security needs. For example, if you choose to go with a database per customer, you can host multiple databases on one RDS instance. And, when you run out of capacity, you can stand up another RDS instance.
• Zero/Minimal Downtime and Seamless Upgrades: Believe it or not, many customers expect that SaaS applications will have zero or very minimal downtime and since these are often managed by the same company who built the application, it should be upgraded seamlessly. The challenge is your application may not have been designed to handle upgrades smoothly, especially if it is a pre-existing application being converted to a SaaS application. There are 2 key aspects to consider a) deploying application bits and files b) handling persistent store upgrades. For rolling out application bits strategies like Blue/Green deployment can be used in which the new release is deployed to a new stack, tested and made live if the rollout was successful. The older stack resources can be decommissioned and reclaimed at a later point. One approach to achieve seamless upgrades is to make the underlying data model n – 1 compatible. What that means is if the release being deployed is of data model version n, it’s data model is backward compatible with the previous data model version (n – 1) thus ensuring that the upgrade will not break it. How can you ensure that? This requires incorporating discipline throughout your development cycle and following certain guidelines, like not deleting any columns, providing necessary upgrade scripts to handle any data migration needs, and so on. And, in the event upgrade is not successful, having support to rollback the upgrade. Now, you can understand this is not only technically challenging as this involves data migration and rollback, but it could also lead to a significantly slower rollout. Hence, you have to evaluate carefully what is reasonable for your application needs and implement the solution accordingly.
• Optimal Cost: As you progress through your application and deployment design, you would be able to come up with multiple approaches to deploy the same application. However, one key aspect to be successful in SaaS is to ensure that you choose the most cost-effective model that not only meets your immediate needs but can cater to at least the near future needs, such as when the demand rises or slows down. You do not want to be spending overtly, but not so less that the application starts choking with very few users. Hence, it is important to strike the right balance. A rule of thumb that I like to follow is when choosing resource sizing always go on the conservative side. And, if the testing reveals higher configuration is needed, then only increase.

DevOps Considerations For SaaS

DevOps is so critical for SaaS that it is worth discussing it separately. The following are some key considerations.

• Continuous Delivery: The DevOps pipelines should be able to take the checked-in code, produce a build from it that then passes through various stages (QA, performance, final go/no go checks, production deploy) in an automated manner. This may involve having multiple pipelines (typically, per stage) and having an uber pipeline that pushes the build through each of these stages. Now, the pipelines may also take some time to develop, but it is a good idea to start defining the contracts for each pipeline so that the consumer pipelines do not need to worry about the details. Eventually, the goal should be to get completely hands-free or as close to it as possible.
• Use Version Control for everything including DevOps changes: For application code, it is generally well understood to use the master branch of the source control. However, it is equally important to do the same for any DevOps changes. For example, when you are rolling out infrastructure changes, these should also be checked into the source control, tested and then pushed to production.
• Agile Infrastructure: To be successful at SaaS, you want to make sure that your infrastructure is agile and can cope up with the changes in demand. As the demand goes up, it can scale appropriate tier and when the demand goes down release the non-required resources. This requires a certain level of experimentation and testing to get to the right balance. For example, you can use AWS auto-scaling capabilities to automatically scale up/down the infrastructure.
• Monitoring and Alerts: The stacks these days have multiple moving parts and troubleshooting could be a bit of a challenge. Hence, it is important to have the right monitoring and alerting mechanisms in place so that when things go wrong, you can troubleshoot the problem more quickly. It is a good idea to leverage from some of the core Cloud monitoring capabilities like centralized logging, alarms, and notifications on errors.

Other Considerations For SaaS

• Planning and Prioritization: Like any other successful project, SaaS projects also require planning and prioritization. While everyone wants to go for goals like “push every check in to production”, see what makes the most business sense and prioritize the important things first. Of course, having a stretch goal is not wrong. But, it is important to get the important things right first. For example, if you do not have a good unit testing and automation coverage and you are trying to push every code change to production, even if you accomplish it, the usefulness is questionable. It can backfire because things can start breaking too quickly in production and then the R&D team will be consumed in handling those.
• Monetization Model: SaaS also impacts the monetization model. While in on-premise you may be fine selling license for a certain amount, in SaaS you may have to rethink what is the most suitable model for your business. Do you want to use a subscription-based model, utilization-based model, a hybrid model, or something else altogether?

Hopefully, you got a better insight into what it takes to design a Cloud-based or SaaS application. It is certainly an enriching experience to see your application that involves so many different aspects, go live in production. As I always say, “Cloud is a journey and not a destination”. So, keep learning and evolving.

Happy designing!
– Nitin

If you liked this post, you will find my AWS Advanced For Developers course helpful that focuses on many such best practices and techniques to design and deploy real-world applications in AWS.

The post How To Design Applications For Cloud (SaaS) appeared first on Cloud Nine Apps.

Whether you are just learning AWS or have been using the amazon web services for some time, you will invariably run into connectivity issues in your deployments. For example, not able to SSH into the EC2 instance, the application tier is not able to talk to the database, and so on. In fact, there may be times when the connectivity was working fine when your stack was deployed, but it broke after some time. I am sure you can relate to at least some of these experiences. In this post, I will talk about how to troubleshoot and resolve some of the commonly encountered connectivity issues in AWS deployments.

Using Telnet For Port Check

Before we start talking about the issues, let’s first learn a simple technique named port check that can be used for troubleshooting. In networking, a port check refers to testing whether a port on a given node is listening or not. For example, if you want to check for the standard SSH port on a machine, you would check port 22. Why is a port check important? It is important because it is one of the most fundamental checks you can do for testing connectivity between two components without even knowing much about the components themselves. To explain this further, if an application running on an EC2 instance is failing to connect to the RDS instance and the port check for the database port fails from the EC2 instance, you can easily confirm that there is some connectivity issue between these two.

You can use the telnet utility for a port check. It is available on most platforms and is often pre-installed or can be easily installed at a later point. In order to do the port check, you will specify a command like the one shown below.

telnet <target-ip-address-or-dns-name> <port>

If you are able to telnet successfully, the port check is successful. Otherwise, it has failed. Simple!

The following screenshot shows a successful port check using telnet on an EC2 instance with the IP address 10.1.0.195 on the SSH port 22.

Here is another screenshot that shows a failed port check. In this case, the telnet connection has simply hung (that is, it is not able to connect successfully). But, you may see other variants like not able to connect, etc.

Common Connectivity Issues In AWS Deployments

Let’s talk about some common connectivity issues in AWS deployments now. Here is a list of such issues.

• Not able to connect to an EC2 instance via SSH.
• An application running on an EC2 instance is not able to connect to the RDS instance.
• The users are not able to access the web application.

These are just some of the commonly encountered issues. But, these represent some commonly observed patterns and you may find other issues that follow the same pattern. So, let’s discuss how to troubleshoot and resolve these.

Not able to connect to an EC2 instance via SSH

If you are not able to SSH into an EC2 instance, you can do a port check using the instance IP or DNS name on the SSH port to see if the connectivity is working at least.

If the port check was successful (that is, you were able to telnet to the SSH port), but you are still not able to connect via SSH, check for the following.

• Use of incorrect SSH user: Although the ec2-user is used quite commonly, certain AMIs use a different user. For example, the default user for the CentOS AMI is centos. You can check out Get Information About Your Instance to find the default user for various AMIs.

However, if the port check failed, the following could be some common reasons.

• Use of incorrect IP address or DNS name for the EC2 instance: This can happen due to a user error or the EC2 instance was rebooted and it’s public IP and DNS name changed. Now, if you have assigned an Elastic IP to the instance, it will not change upon reboot. But, Elastic IPs are expensive and used only for important instances typically. So, make sure to check the IP/DNS name is correct.
• A Firewall is blocking the SSH connection: At times, the corporate InfoSec team may block SSH connectivity to public IP addresses. This is typically done to avoid the scenario where a hacker can take advantage of an SSL vulnerability to hack into the corporate network. You can typically do some initial troubleshooting for this by using the SSH verbose option (-v or -vvv) when establishing the SSH connection. If that’s the case, you will need to work with your InfoSec team on the resolution. One possible solution is to assign an Elastic IP to your EC2 instance and get an exception from them to allow SSH to this IP address. Another potential firewall issue could be the OS firewall is blocking the SSH connection (such as the iptables configuration). To fix this you will have to modify the firewall configuration (preferably) or turn the firewall off.
• Incorrect Security Group configuration: This is quite common especially in manual deployments wherein either the Security Group that permits SSH connectivity has not been assigned to the EC2 instance or it does not have the proper ingress rule. So, double-check the Security Group assignment and ingress rules. Remember that you can assign multiple security groups to an EC2 instance and you can change the Security Group assignment post instance creation as well. The following screenshot shows a sample security group assignment to the EC2 instance.
  

An application running on an EC2 instance is not able to connect to the RDS instance

To troubleshoot this further, you can do a port check from the EC2 instance to the RDS instance port and see if that works.

The following screenshot shows a successful telnet port check to an RDS instance.

In that case, check for the following.

• Use of incorrect RDS instance name or port: Check your application configuration to see if it has the correct RDS instance name and port value.

If the port check failed, these could be some common reasons.

• Use of incorrect RDS instance name or port: Double-check the instance name and port to ensure these are correct.
• Is the RDS instance up? RDS supports the shutdown of instances. This is often useful for cost control purposes when the instance is not in use. For example, shutting down a development RDS instance during the weekends. So, check if the RDS instance is available.
• Incorrect Security Group configuration: Check whether the Security Group assigned to the RDS instance has correct ingress rules to permit connectivity from the EC2 instance. This would typically involve ensuring the ingress rule has the correct source subnet (which should be your EC2 subnet) and target (database) port specified. The following screenshot shows ingress rule entries for MySQL/Aurora RDS instance.
  
• Missing VPC Peering Configuration: Often resources like RDS instances are shared between application teams to reduce the cost. In such cases, the RDS instance is hosted in a different VPC (say, common VPC) and the applications are typically deployed in their own VPCs. VPC Peering is used to connect the application VPC with the common VPC so that these can talk to the RDS instance. If the port check failed, check the VPC Peering to ensure a Peering Connection has been made between these two VPCs and route table entries have been created in both these VPC route tables to route traffic to their respective subnets. For example, the screenshot below shows adding an entry in the common VPC route table for the application VPC subnet 10.1.0.0/16 that is routed via the Peering Connection pcx-09fbbdba113f3bb44. Similarly, an entry will need to be created in the application VPC’s route table for the common VPC subnet as the destination.
  

Although we talked about RDS instance here, these troubleshooting techniques can be used for connectivity issues for other application stack components, such as Consul, Elasticsearch, and so on.

The users are not able to access the web application

Again, a port check can be used here to see if the connectivity is working.

If the port check is successful, the following could be some common reasons.

• Application issue: Your application may be having issues. For example, the application stack may not be up or having other issues. So, check the application logs for any potential errors.
• Are you using a load balancer? For a deployment that is using a load balancer (LB), check for the instance health in load balancer setup to ensure that LB is able to use these to serve traffic. Note that LB will only serve traffic to healthy instances. If any of the instances are showing unhealthy, check their logs to troubleshoot further.

If the port check failed, these could be some common reasons.

• Use of incorrect IP address or DNS name: Ensure that the correct IP address/DNS name is being used to access the application.
• Use of incorrect port: Is your application running on a non-standard web port (that is, a port other than 80 (for HTTP) and 443 (for HTTPS))? For example, if you are using tomcat, the application may be running on port 8080 instead. So, check your configuration and ensure that the correct port is being used by the users.
• Incorrect Security Group configuration: Check whether the correct Security Group has been assigned to the web tier resource(s) and verify it’s ingress rules to ensure it permits connectivity to the correct port. For example, the screenshot below shows an ingress rule to permit traffic on port 8080 from any IP address.
  

In this post, we covered one of the most commonly used port check technique. It can be used in combination with other tools and techniques to effectively troubleshoot and identify the root cause. Troubleshooting connectivity issues can become a challenging task. Hence, it is important to spend some time and familiarize yourself with these tools and techniques in advance so that you are better prepared when issues arise.

Be a smart troubleshooter!
– Nitin

If you liked this post, you will find my AWS Advanced For Developers course helpful that focuses on many such best practices and techniques to design and deploy real-world applications in AWS.

The post How To Troubleshoot Connectivity Issues In AWS Deployments? appeared first on Cloud Nine Apps.

Automating the AWS deployments has been a key driver to ensure consistent and reliable deployments. Whether it is zero-touch deployments, immutable architecture or continuous delivery (CD), automated deployments are critical to successful delivery. And, this is not just about deploying the underlying infrastructure. It is also equally applicable to the application stacks as well. AWS CloudFormation is a key service when it comes to automating AWS deployments. Be it a simple stack with a couple of resources or complex stacks that are deployed to multiple AWS regions and accounts, CloudFormation provides several useful capabilities like reusable deployment templates, powerful CLI, automatic change detection, rollback, resource dependency management, parallel deployment, and many more. In this post, we will learn some basics along with an actual example and see why is CloudFormation an Architect’s best friend!

What is CloudFormation (CFN)?

First things first. CloudFormation (CFN) is a service from the AWS portfolio that delivers Infrastructure as a Code. It offers the following key capabilities.

• Automate complete infrastructure setup: Manage the complete deployment recipe of your deployment by specifying resources and their configuration. A CFN deployment is also referred to as a stack.
• Focus on “what” and not “how”: CFN is developed in the form of text-based templates (a.k.a. CFN template) in which the majority of times you specify “what to do”. For example, when you create an EC2 instance, you simply specify it’s configuration (like the instance type, storage size, etc) and not the actual commands to create the resource. This makes the templates relatively small in size and helps you focus on what you want to accomplish. Having said that it does offer (limited) scripting capabilities for common needs, such as string manipulation.
• Consistency with speed: CFN offers features like parallel deployment for faster turnaround. At the same time, the templates can be designed to ensure all deployments are consistent and avoid manual configurations altogether.
• Manage the complete lifecycle of the stack: You can create/update/delete stacks.
• Version Control Infrastructure Releases: Yes, that’s possible! Since CFN templates are text-based documents (JSON or YAML), you can simply check these into your version control and use the same standard best practices that you are used to for a typical application release. In fact, I highly recommend to only deploy from the “master” branch to production to ensure only validated changes are deployed.
• Apart from these, it offers several other useful capabilities like dependency management between resources (dependency resources are deployed before and deleted after the dependent resources), delta detection when making updates, an automated rollback of failed deployment, and so on.
• CFN offers a powerful CLI and SDK support for programmatic integrations. This is extremely useful for DevOps and automation purpose.

CloudFormation Core Concepts

Let’s talk about some basic CFN constructs now. Keep in mind that the key idea with CFN is to have reusable templates that can be used to deploy multiple stacks typically.

• Parameters: These are used to take user inputs and capture variables that can change between deployments from the same template. A parameter has a type (such as String) and can optionally have validation associated. AWS offers pre-defined parameters, known as Pseudo Parameters, for some useful values, such as the deployment account ID, the target region, and so on.
• Mappings: These are used to capture derived information. For example, you can specify resource attributes for different deployment type (development vs production).
• Conditions: A condition can be used to specify the conditional creation of a resource or use of a resource property.
• Resources: The resources and their configuration specified using properties.
• Outputs: A CFN template can produce outputs to provide relevant information. For example, an RDS template can provide the database instance connection information.
• Metadata: Useful metadata can be specified in the CFN template that can be consumed by other tools, such as CloudFormation Designer and automation tools. For example, CFN offers helper scripts that can use the metadata to install the software.
• Intrinsic Functions: CFN provides several useful intrinsic functions for common computing needs, such as string manipulation, resource lookup, etc. These are evaluated at deployment time.

CFN Template Example

Let’s walk through an example to understand these concepts better.

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "An EC2 instance.",
  "Parameters": {
    "InstanceName": {
      "Description": "The instance name.",
      "Type": "String"
    },
    "DeploymentType": {
      "Description": "The deployment type.",
      "Type": "String",
      "AllowedValues": ["Dev", "QA", "Prod"],
      "Default": "Dev"
    },
    "Subnet": {
      "Description": "The subnet for the EC2 instance.",
      "Type": "AWS::EC2::Subnet::Id"
    },
    "SecurityGroups": {
      "Description": "The Security Groups for the EC2 instance.",
      "Type": "List"
    },
    "KeyPair": {
      "Description": "The key pair name to use to connect to the EC2 instance.",
      "Type": "String"
    }
  },
  "Mappings": {
    "Globals": {
      "Constants": {
        "ImageId": "ami-0b898040803850657",
        "AssignPublicIP": "true"
      }
    },
    "DeploymentTypes": {
      "Dev": {
        "InstanceType": "t2.small",
        "StorageSize": "20"
      },
      "QA": {
        "InstanceType": "t2.small",
        "StorageSize": "30"
      },
      "Prod": {
        "InstanceType": "t2.medium",
        "StorageSize": "50"
      }
    }
  },
  "Resources": {
    "EC2Instance": {
      "Type": "AWS::EC2::Instance",
      "Properties": {
        "ImageId": {"Fn::FindInMap": ["Globals", "Constants", "ImageId"]},
        "InstanceType": {"Fn::FindInMap": ["DeploymentTypes", {"Ref": "DeploymentType"}, "InstanceType"]},
        "NetworkInterfaces": [{
          "DeviceIndex": "0",
          "SubnetId": {"Ref": "Subnet"},
          "AssociatePublicIpAddress": {"Fn::FindInMap": ["Globals", "Constants", "AssignPublicIP"]},
          "GroupSet": {"Ref": "SecurityGroups"}
        }],
        "BlockDeviceMappings": [{
          "DeviceName": "/dev/sdm",
          "Ebs": {
            "VolumeType": "gp2",
            "VolumeSize": {"Fn::FindInMap": ["DeploymentTypes", {"Ref": "DeploymentType"}, "StorageSize"]},
            "DeleteOnTermination": "true"
          }
        }],
        "KeyName": {"Ref": "KeyPair"},
        "Tags": [{"Key": "Name", "Value": {"Ref": "InstanceName"}}]
      },
      "Metadata": {
        "AWS::CloudFormation::Designer": {
          "id": "d0aacb0c-1b2c-452c-baf0-b283b0ba4a1a"
        }
      }
    }
  },
  "Outputs": {
    "PublicDNSName": {
      "Description": "The public DNS name.",
      "Value": {"Fn::GetAtt": ["EC2Instance", "PublicDnsName"]}
    },
    "PublicIP": {
      "Description": "The instance public IP address.",
      "Value": {"Fn::GetAtt": ["EC2Instance", "PublicIp"]}
    }
  },
  "Metadata": {
    "AWS::CloudFormation::Designer": {
      "d0aacb0c-1b2c-452c-baf0-b283b0ba4a1a": {
        "size": {
          "width": 60,
          "height": 60
        },
        "position": {
          "x": 546,
          "y": 153
        },
        "z": 0
      }
    }
  }
}

This template deploys an EC2 instance. Following are the details.

• The template format version and description are specified in the beginning.
• The Parameters specifies deployment inputs that can change across stacks deployed from this template.
  • The InstanceName parameter captures the EC2 instance name.
  • The DeploymentType parameter is interesting. I call this a logical parameter. It simplifies the user experience by avoiding to ask unnecessary details from the user. At the same time, it allows customizing different EC2 instance deployments from the template by choosing from {Dev, QA or Prod} values.
  • The Subnet parameter is used to specify the EC2 instance subnet.
  • The SecurityGroups parameter specifies the security groups to be associated with the instance.
  • The KeyPair parameter takes the SSH keypair name that will be used to connect to the EC2 instance.
• The Mappings section specifies two top-level maps – 1) Globals and DeploymentTypes maps. And, each of these maps contains one or more maps that store data.
  • The Globals map is used here to capture useful constants – the AMI ID to use and whether to assign public IP to the EC2 instance. Why would you have such a map? It’s for ease of maintenance. Tomorrow if you have to make a change, simply update the constants.
  • The DeploymentTypes map specifies EC2 instance properties based on the deployment type. For example, for the Dev deployment, we would like to use InstanceType as t2.small.
• It creates a single resource – an EC2 instance. This also shows why CFN is more about “what” than “how”. See we are only specifying resource configuration and not the commands to create the instance. Let CFN do it’s magic!
  • The ImageId property is set by looking up the Globals->Constants->ImageId value and using the Fn::FindInMap intrinsic function.
  • The InstanceType property is set by looking up the map for the DeploymentType. This uses the Ref intrinsic function to get the DeploymentType parameter value and then passes it to the Fn::FindInMap intrinsic function to retrieve the property value.
  • The NetworkInterfaces property specifies a single network interface with SubnetId set to the Subnet parameter value, AssociatePublicIpAddress set to the Globals->Constants->AssignPublicIP value, and GroupSet set to the SecurityGroups parameter value.
  • The BlockDeviceMappings property specifies a single EBS volume of type gp2 and size by looking up the DeploymentTypes-><DeploymentType>->StorageSize value. For example, for a development stack, the DeploymentTypes->Dev->StorageSize value will be used.
  • The KeyName property is set using the KeyPair parameter value.
  • The Name tag is set to the InstanceName parameter value.
• The Outputs section outputs the public DNS name and IP information using the Fn::GetAtt intrinsic function to retrieve the PublicDnsName and PublicIp attributes of the EC2 instance, respectively.
• The Metadata sections of this template contain the user interface data for the CFN Designer tool.

Conclusion

We covered a lot of ground here. But, this is just a highlight of how powerful CFN is. It offers many more capabilities like resource dependency management, nested stacks that make it easy to dploy a hierarchy of stacks, StackSets that allow cross-region and even cross-account deployment, and more. In fact, if you have a resource that is not directly managed by AWS (or not supported by CFN), you can still manage it using CFN via a Custom Resource. So, you see the possibilities are endless and we have only scratched the surface. But, you know by now that if you are an Architect or someone who is involved in AWS deployments, CloudFormation is a tool that can help you tremendously in automating your deployments.

Happy deploying!
– Nitin

The post AWS CloudFormation – An Architect’s Best Friend appeared first on Cloud Nine Apps.

We are extremely glad to announce the launch of our AWS CloudFormation Deep Dive course. It focuses on using CloudFormation for automating your AWS deployments and continues the legacy of our hands-on courses that focus on real-world deployments. AWS CloudFormation is a key service when it comes to automating AWS deployments. Be it a simple stack with a couple of resources or complex stacks that are deployed to multiple AWS regions and accounts, CloudFormation provides several useful capabilities like reusable deployment templates, powerful CLI, automatic change detection, rollback, resource dependency management, parallel deployment, and many more. This course will set a solid foundation for anyone working with AWS deployments in understanding CloudFormation core concepts to hands-on experience in deploying simple to complex stacks along with several best practices and tips.

Course Overview

Course Outline

Following is an outline of the course.

• Overview of AWS CloudFormation (CFN)
• Best practices and tips for using CFN in real-world deployments
• Deep dive into CFN with detailed examples
• Advance CloudFormation topics like Nested Stacks, StackSets, Custom Resources and Macros

Intended Audience

• Application Architects
• Cloud Deployment Designers and Practitioners
• DevOps
• DevOps Engineers
• QA
• Operations

Click here to learn more and enroll.

The post Announcing Launch of AWS CloudFormation Deep Dive Course appeared first on Cloud Nine Apps.

AWS CloudWatch plays a critical role for most deployments. It provides several capabilities from log management to generating alarms, dashboards and handling events. In this tutorial, we will cover an overview of CloudWatch, it’s capabilities, basic usage, and some best practices.

CloudWatch Overview

CloudWatch Key Capabilities

CloudWatch offers the following key capabilities.

• Log Management:
  • You can publish logs to CloudWatch – both application and any system logs.
  • You can retrieve logs and also perform basic analysis using the CloudWatch console, such as searching for error messages.
  • Several services provide integration with CloudWatch, such as lambda and API Gateway. For EC2, you can use the CloudWatch agent to publish logs to CloudWatch.
  • CloudWatch is often used for log consolidation from several different components. For example, you may have logs coming from your EC2 servers, API Gateway, and so on.
  • CloudWatch is designed for scale and you can store a huge amount of logs. In fact, it does not purge out logs by default unless you set an expiry. Hence, you must make sure to set out an appropriate expiry on the Log Groups to avoid getting a huge bill.
• Metrics
  • CloudWatch supports capturing metrics that can be later used for various purposes, such as reporting or taking an action. It offers several out-of-the-box metrics. For example, you could monitor the CPU utilization of your EC2 instances.
  • CloudWatch also offers AWS Custom Metrics. These metrics are provided by AWS tools, such as the CloudWatch Agent for EC2 that can report AWS Custom Metrics like memory and disk utilization.
  • Custom Metrics can also be published to CloudWatch. These are often used to take advantage of CloudWatch reporting and action capabilities. For example, you could have a metric to capture errors in the logs and trigger an email when that metric count is greater than 1. A custom metric need not always be technical. You could also use it for business purpose, such as to report a transaction above a certain monetary value.
• Alarms
  • An alarm is used to take one or more actions on a metric, such as send an email.
  • AWS offers various ways to evaluate alarm conditions, such as average, count, sum, etc.
• Monitoring
  • CloudWatch offers several monitoring capabilities from near real-time monitoring of metrics to user-created dashboards, which are often used to monitor critical metrics.
• Events
  • Events use Rules to handle changes to your AWS resources. For example, when an EC2 instance is started or stopped.
  • An Event Rule can also be scheduled, such as to periodically refresh data.

CloudWatch Logs Overview

Let’s understand some details about CloudWatch Logs.

• CloudWatch organizes logs into Log Groups. Think of a Log Group as a folder that represents a micro-service or component in your deployment. For example, you may have a log group for your application tier.
  Note: As you can see in the screenshot below, the default expiry is set to “Never Expiry”. Hence, it is a good idea to always set this to a value that meets your deployment needs. For example, for production, you may want to keep logs up to 2 weeks old. But, for development and QA you may be fine to purge out logs that are more than 3 days old. So, configure a value as appropriate for your deployment.
  
• A Log Group contains one or more Log Streams. Think of a Log Stream as a log file. For AWS services like EC2, typically there will be a Log Stream per EC2 instance. But, for other services like lambda, a Log Stream is for a chunk of time.
  
• Finally, we have the actual log content within a Log Stream.
  
  • A log entry is always in the UTC time. This way you do not have to convert time into server time. This is especially useful when you have logs from different regions and multiple components.
  • CloudWatch maintains a timestamp of when the entry was published.
  • You can choose a timeframe to view the logs.
  • CloudWatch console offers basic search capabilities for analysis purpose.

CloudWatch Metrics Overview

CloudWatch offers several out-of-the-box metrics. In addition, custom metrics can also be created. Here are the key points about metrics.

• You can look at the currently available metrics from the Metrics view.
  
• You can drill down into a specific metric. For example, the screenshot below shows the CPUUtilization metric for a specific EC2 instance. This way you can do near real-time monitoring of the metric.
  Note: You can adjust the time window based on the needs and you can also save this graph to a dashboard.
  

CloudWatch Alarms Overview

An Alarm lets you take one or more actions when a given metric threshold is breached and also when it comes back to normal. For example, you may want to send out an alert to an infrastructure team when the CPU utilization of a critical production EC2 instance is high.

Following are the steps to create an alarm.

1. Choose a metric Category to help get to the appropriate metric.
   
2. Select the Metric. In the screenshot below, we have selected the CPUUtilization metric for a specific EC2 instance.
   
3. Specify the Alarm details.
   
   • GIve a logical name. Alarms are tied to specific resources. Hence it is useful to give a reference to the resource (or at least type of resource) in the alarm name.
   • Give a description for your reference.
   • Specify the threshold. Here we are specifying a threshold of 75% for 3 consecutive data points. That is when the CPUUtilization >= 75% for 15 minutes.
   • Specify how to handle missing data. This typically happens in the initial state of alarm when data is not yet available. But, it could also happen for other metrics where the reporting may miss a cycle (such as a metric published using a CloudWatch agent).
   • Specify the actions. In this case, we are simply stopping the instance.
4. Once the alarm has been defined, you can see it on the list of alarms. Initially, the alarm will show INSUFFICIENT_DATA until it has gathered enough metric data to go to the normal state or the alarm state.
   

CloudWatch Events Overview

CloudWatch Events allow handling of changes to resources, such as when an EC2 instance changes state. Let’s see how to handle a simple Event.

1. In order to handle an Event, create an Event Rule.
   
   • Choose an Event Source. It could be an event on an AWS resource or a schedule based execution (similar to a cron job). In this screenshot, we are taking EC2 instance state change events and specifically selecting the running/stopped/terminated.
   • The Event Pattern Preview shows a sample event that the event handler can expect. This is helpful as you can copy this as a test event to test out the event handler.
   • Specify Targets that would handle the event. You can specify multiple targets. In this case, we have specified a lambda function. When the event occurs, the lambda will be passed an event similar to the one shown in the preview.
2. Give a logical name to the rule and create it.
   
3. Once the rule is created it will show on the rules list.
   

CloudWatch Dashboards Overview

CloudWatch Dashboards make it convenient to monitor commonly observed metrics. These are perfect for Operations teams usage and even for development and testing teams. For example, every time you run performance tests, you would want to monitor and grab the relevant widgets from the dashboard. Following is a sample dashboard that shows memory usage.

• You can create multiple dashboards, such as one per application or even component.
• You can add widgets to an existing dashboard from the Dashboards view or from the Metrics view. A widget shows a chart.
• When you add a widget try keeping metrics with the same unit only so that the dashboard is easy to read.

CloudWatch Best Practices

Following are some key best practices to follow for CloudWatch.

• Use an appropriate naming convention for CloudWatch Log Groups and Log Streams.
• Always set an appropriate expiry for CloudWatch Log Groups. Otherwise, the consolidated logs can lead to a huge bill.
• Like many AWS services, CloudWatch also offers a free tier. Take advantage of it. At the same time keep a watch on the resources to not exceed. Often times certain resource creation processes may create CloudWatch resources behind the scenes, such as Log Groups, alarms, etc. Make adjustments to these as needed and delete ones that you don’t need.
• Use Dashboards for frequently monitored metrics.
• Use metrics with the same unit on a dashboard widget.

Happy learning!
– Nitin

The post Not Just Another AWS CloudWatch Tutorial appeared first on Cloud Nine Apps.

Simple Storage Service (S3) was one of the initial set of AWS services and has been increasingly popular. It offers object-based storage with low latency, high durability, and availability. In this tutorial, we will cover an overview of S3, how to create a bucket, it’s basic usage and some tips.

S3 Overview

Following are the key capabilities offered by S3.

• It is a Global service.
• It offers object-based storage. Think of it as a file share in the cloud. It cannot be used for a filesystem (such as for EC2).
• S3 organizes objects into buckets.
  • A bucket belongs to an AWS region.
  • A bucket name is globally unique. That’s right! Remember S3 is a global service.
  • An S3 bucket can be public or private. It is recommended to avoid making buckets public. If you need to share a bucket, there are other ways of sharing, which we will discuss later in this tutorial.
  • It can store multiple objects.
  • A folder hierarchy can be created to further organize the objects.
• S3 supports object lifecycle management capabilities like versioning and storage tiering.
  • Versioning allows maintaining and using multiple versions of an object.
  • Storage tiering is a popular concept in storage domain to move an object to a more aptly priced storage (typically cheaper) based on the lifecycle and access needs. For example, for data that does not need to be frequently accessed, it could be moved to Amazon Glacier, which is cheaper than keeping it in S3. This is accomplished via lifecycle policies.
• S3 offers various levels of availability and durability based on the storage class. For example, the S3 Standard storage class (the default) offers high availability and durability for frequently accessed data, whereas, S3 Standard-IA (Infrequent Access) offers reduced availability as the data is intended to be infrequently accessed. S3 pricing varies based on the storage class.

Getting Started With S3

Create an S3 Bucket

Let’s see how to create a bucket in S3.

1. Go to the S3 Console and click on Create bucket.
   
2. Specify a universally unique bucket name and choose a region in which the bucket will be hosted. Typically, you would choose the region based on your application or consumers who would be accessing this bucket. You can also choose to copy settings from another bucket here.
   Note: If you are practicing the instructions covered in this tutorial, make sure to choose a different bucket name than the one shown here since it has to be globally unique.
   
3. You can set additional properties on the bucket.
   
   • A couple of interesting ones are versioning and encryption. Encryption is used to encrypt the data at rest, which is often required for security and compliance purpose.
     • S3 offers various types of encryptions from Server-Side Encryption with S3 managed keys (SSE-S3) to customer-managed keys.
     • Of course, you can also use client-side encryption in which first the files are encrypted by the client and then uploaded to S3.
4. Set the bucket permissions to specify who can access it. It is recommended to not make the bucket public. Also, keep the original permission here for the owner only. We will see later in this tutorial how to share a bucket with others.
   
5. Review the details and create the bucket.
   
6. Once the bucket has been created, it will appear on the list of buckets.
   

Upload a File to S3

We can now upload a file to our S3 bucket using the following steps.

1. Click on the bucket and select Upload.
   
2. Add the files to upload.
   
3. Specify the permissions. Here you can choose to override the permissions just for the object(s) being uploaded. In most cases, you’ll just go with the defaults.
   
4. Set various properties based on the needs, such as Storage class and Encryption.
   
5. Review the details and upload the file.
   
6. The file will now show up under the bucket.
   
7. You can click on the file to get more details and perform actions on it. You can find various details like size and the download link.
   

S3 Bucket and Object Management

Once you have the buckets and objects in S3, here are some commonly performed operations.

• Get objects
• Update objects
• Delete objects
• Add more objects to buckets
• List objects in a given bucket
• Delete bucket

Understanding S3 Bucket Policies

A bucket policy enables access and type of actions permitted on a given S3 bucket. As such, these are applied at the bucket level and apply to all objects within the bucket. Following are some key capabilities offered by the bucket policies.

• Grant access to specific IAM users.
• Share a bucket with one or more AWS accounts without making it public.
• Restrict access to the bucket to a certain domain or set of IP addresses. A use case for this would be if you are storing content for your website on S3 and want to make sure that only requests from your site are able to access the S3 bucket objects.

Let’s take a look at a sample bucket policy.

{
  "Version":"2012-10-17",
  "Statement":[
    {
      "Sid":"AllowAccessFromSpecificIP",
      "Effect":"Allow",
      "Principal":"*",
      "Action": "s3:GetObject",
      "Resource":"arn:aws:s3:::cloudnineapps-demo/*",
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": [
             "your-public-ip-address/32"
          ]
        }
      }
    }
  ]
}

A bucket policy has a structure similar to an IAM Policy. This policy enables only a specific IP address to get the objects from the cloudnineapps-demo bucket. Let’s review its details.

• The Principal specifies the target for access. In this case, it applies to all.
• We are only providing permission to get objects from the bucket.
• The Resource specifies the target bucket.
• The Condition specifies additional criteria. In this case, we are allowing access from a specific IP address that should be provided in place of the text your-public-ip-address.

S3 Best Practices

Following are some key best practices to follow for S3.

• Avoid making S3 buckets public.
• Use bucket policies for controlling access to the bucket (including sharing with other AWS accounts).
• Use Server-Side Encryption to encrypt the data at rest.
• Make it a practice to periodically review the buckets and their content to ensure any stale content is purged out to save cost.

Happy learning!
– Nitin

The post Not Just Another AWS S3 Tutorial appeared first on Cloud Nine Apps.