This chapter is from the book
This chapter is from the book
This chapter is from the book
Automating AWS Architecture
Many systems have been put in place over the years to help successfully manage and deploy complicated software applications on complicated hardware stacks.
If you look at the AWS cloud as an operating system hosted on the Internet, the one characteristic of AWS that stands above all others is the level of integrated automation used to deploy, manage, and recover AWS services. There is not a single AWS service offered that is not heavily automated for deployment and in its overall operation. When you order a virtual private network (VPN), it’s created and available in seconds. If you order an Elastic Compute Cloud (EC2) instance, either through the AWS Management Console or by using the AWS command-line interface (CLI) tools, it’s created and available in minutes. Automated processes provide the just-in-time response you want when you order cloud services.
AWS services are being changed, enhanced, and updated 24/7, with features and changes appearing every day. AWS as a whole is deployed and maintained using a combination of developer agility and automated processes; it is able to move quickly and easily with a partnership of developers, system operations, project managers, network engineers, and security professionals working together from the initial design stages, through the development process, to production and continual updates.
AWS wasn’t always so automated and regimented. In the early days, Amazon was a burgeoning online e-commerce bookseller. Not so very long ago, you would order a virtual machine from a cloud provider and wait several days for an email to arrive, telling you that your service was ready to go. As the Amazon e-commerce site became more popular, new problems appeared in the realm of scaling online resources to match customers’ needs. Over time, Amazon developed rules for all developers, mandating that each underlying service that supported the Amazon store must be accessible through a core set of shared application programming interfaces (APIs) that were shared with all developers; each service needed to be built and maintained on a common core of compute and storage resources.
Amazon built and continues to build its hosting environment using mandated internal processes, which can be described as a mixture of the following:
ITIL: Information Technology Infrastructure Library (ITIL) is a framework of best practices for delivering IT services.
Scrum: Scrum is a framework in which a development team works together to manage product development.
Agile: Agile is a software development cycle in which developers plan, design, develop, test, and evaluate as a team with open communications.
DevOps: DevOps is a continuation of the Agile framework that features full collaboration among the development and operations teams.
Many people at AWS are working together in an effective manner to make hundreds of changes to the AWS hardware and software environment every month. In addition, all AWS services are being monitored, scaled, rebuilt, and logged through completely automated processes. Amazon avoids using manual processes, and your long-term goal should be for your company to do the same.
As your experience with AWS grows, you’re going to want to start using automation to help run and manage your day-to-day AWS operations and to solve problems when they occur. There are numerous services available that don’t cost anything additional to use aside from the time it takes to become competent in using them. This might sound too good to be true, but most of Amazon’s automation services are indeed free to use; you are charged only for the AWS compute and storage resources that each service uses.
Automation services will always manage your resources more effectively than you can manually. At AWS, the automation of infrastructure is typically called infrastructure as code (IAC). When you create resources using the AWS Management Console, in the background, AWS uses automated processes running scripts to finish the creation and management of resources.
Regardless of how you define your own deployment or development process, there are a number of powerful tools in the AWS toolbox that can help you automate procedures:
CloudFormation: For stack-based architecture deployments
Service Catalog: To help secure CloudFormation templates
Elastic Beanstalk: For deploying applications and infrastructure together
Automating Infrastructure with CloudFormation
The second and third times you deploy EC2 instances using the AWS Management Console, you will probably not perform the steps the same way you did the first time. Even if you do manage to complete a manual task with the same steps, by the tenth installation your needs will have changed or better options will have become available. A manual process rarely stays the same over time. To make changes easier, you can automate even the simplest manual processes at AWS.
If you peek under the hood at any management service running at AWS, you’ll find the process command set driven by JavaScript Object Notation (JSON) scripts. At the GUI level, you use the Management Console to fill in the blanks; when you click Create, JSON scripts are executed in the background to carry out your requests. AWS’s extensive use of JSON is similar to Microsoft Azure’s extensive use of PowerShell; at AWS, JSON scripts are used internally for many tasks and processes; creating security policy with Identity and Access Management (IAM) and working with CloudFormation are two examples that you will commonly come across.
If you use Windows EC2 instances at AWS, you can also use PowerShell scripting. Both Microsoft Azure and AWS heavily rely on automation tools for everyday deployments. In the background, Azure relies heavily on PowerShell scripting and automation.
CloudFormation is an AWS-hosted orchestration engine that works with JSON templates to deploy AWS resources on demand or on predefined triggers (see Figure 3-13). AWS uses CloudFormation extensively, and so can you. More than 300,000 AWS customers use CloudFormation to manage deployment of just about everything, including all infrastructure stack deployments. A CloudFormation template declares the infrastructure stack to be created, and the CloudFormation engine automatically deploys and links the needed resources. You can add additional control variables to each CloudFormation template to manage and control the precise order of the installation of resources.
)
FIGURE 3-13 The CloudFormation Console
Consider this comparison of the manual AWS deployment process and the automated process that starts with CloudFormation:
Time spent: Over time, running manual processes at AWS becomes a big waste of time for the human carrying out the process. In the past, maintaining manual processes such as building computer systems and stacks provided job security; these days, it’s just not a prudent way to deploy production resources. For one thing, there are many steps in a manual process—which means there are many places the process can go wrong. Every CloudFormation deployment does take time, but it takes much less time than a manual process because each step in a CloudFormation script is essential. There are no wasted steps, and all steps are in the proper order. Over time, executing an automated process to build EC2 instances will save you hours or even weeks of time; the CloudFormation process runs in the background, allowing you to do something else. CloudFormation can also perform updates and deletions for existing AWS resources.
Security issues: Humans make mistakes, and manual changes can end up being huge security mistakes due to a lack of oversight. CloudFormation templates can be secured and controlled for usage by specific IAM users and groups. Templates also carry out the same steps every time they are executed, which prevents the fat-finger mistakes that humans make. The AWS Service Catalog service integrates with CloudFormation to mandate which users or accounts can use CloudFormation templates to build infrastructure stacks.
Documentation: It is difficult to document manual processes if they constantly change, and it can be difficult to find time to create documentation. CloudFormation templates are readable, and when you get used to the format, they are actually self-documenting. Again, there are no wasted steps in a CloudFormation script; what is described is exactly what is deployed. If mistakes are found in a CloudFormation script during deployment, all changes that have been carried out are reversed.
Repeatability: If you’re lucky, you can repeat your manual steps the same way every time. However, trying to do so is a waste of valuable time in the long run. With a CloudFormation script, you can deploy and redeploy the listed AWS resources in multiple environments, such as separate development, staging, and production environments. Every time a CloudFormation template is executed, it repeats the same steps.
Cost savings: CloudFormation automation carries out stack deployments and updates much faster than manual processes ever could. In addition, CloudFormation automation can be locked down using a companion service called AWS Systems Manager, discussed later in this chapter, to ensure that only specific IAM users and groups can access and execute specific CloudFormation deployment tasks.
CloudFormation Components
CloudFormation works with templates, stacks, and change sets. A CloudFormation template is an AWS resource blueprint that can create a complete application stack or a single stack component, such as a VPC network complete with multiple subnets, Internet gateways, and NAT services all automatically deployed and configured. You can create a template type called a change set to help visualize how proposed changes will affect AWS resources deployed by a CloudFormation template.
CloudFormation Templates
Each CloudFormation template is a text file that follows either JSON or YAML formatting standards. CloudFormation responds to files saved with JSON, YAML, and .txt extensions. Each template can deploy or update multiple AWS resources or a single resource, such as a VPC or an EC2 instance. Example 3-1 shows a CloudFormation template in JSON format, and Example 3-2 displays the same information in YAML format. It’s really a matter of personal preference which format you use. When creating CloudFormation templates, you might find YAML easier to read, which could be helpful in the long term. YAML seems more self-documenting because it’s easier to read.
Example 3-1 CloudFormation Template in JSON Format
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "EC2 instance",
"Resources" : {
"EC2Instance" : {
"Type" : "AWS::EC2::Instance",
"Properties" : {
"ImageId" : "ami-0ff8a91497e77f667",
"InstanceType" : "t1.micro"
}
}
}
}
Example 3-2 CloudFormation Template in YAML Format
AWSTemplateFormatVersion: '2010-09-09' Description: EC2 instance Resources: EC2Instance: Type: AWS::EC2::Instance Properties: ImageId: ami-0ff8a91497e77f667
CloudFormation templates can have multiple sections, as shown in Example 3-3. However, the only mandatory section is Resources. As with any other template or script, the better the internal documentation, the more usable a CloudFormation template is—for the author as well as other individuals. It is highly recommended to use the Metadata section for comments to ensure that users understand the script.
Example 3-3 Valid Sections in a CloudFormation Template
"AWSTemplateFormatVersion": "version date",
"AWSTemplateFormatVersion": "2010-09-09"
<TemplateFormatVersion: Defines the current CF template version>
"Description": "Here are the additional details about this template
and what it does",
<Description: Describes the template: must always follow the version
section>
"Metadata": {
"Metadata" : {
"Instances" : {"Description : "Details about the instances"},
"Databases" : {"Description : "Details about the databases"}
}
},
<Metadata: Additional information about the resources being deployed
by the template>
"Parameters" : {
"InstanceTypeParameter" : {
"Type" : "String" ,
"Default" : "t2.medium" ,
"AllowedValues" : ["t2.medium", "m5.large", "m5.xlarge"],
"Description" : "Enter t2.medium, m.5large, or m5.xlarge.
Default is t2.medium."
}
}
<Parameters: Defines the AWS resource values allowed to be selected
and used by your template>
"Mappings" : {
"RegionMap" : [
"us-east-1 : { "HVM64 : "ami-0bb8a91508f77f868"},
"us-west-1 : { "HVM64 : "ami-0cdb828fd58c52239"},
"eu-west-1 : { "HVM64 : "ami-078bb4163c506cd88"},
"us-southeast-1 : { "HVM64 : "ami-09999b978cc4dfc10"},
"us-northeast-1 : { "HVM64 : "ami-06fd42961cd9f0d75"}
}
}
<Mappings: Defines conditional parameters defined by a "key"; in this
example, the AWS region and a set of AMI values to be used>
"Conditions": {
"CreateTestResources": {"Fn::Equals" : [{"Ref" : "EnvType"}, "test"]}
},
<Conditions: Defines dependencies between resources, such as the
order when resources are created, or where resources are created.
For example, "test" deploys the stack in the test environment >
CloudFormation Stacks
AWS has many sample CloudFormation templates that you can download from the online CloudFormation documentation, as shown in Figure 3-14, and deploy in many AWS regions. A CloudFormation stack can be as simple as a single VPC or as complex as a complete three-tier application stack, complete with the required network infrastructure and associated services.
)
FIGURE 3-14 AWS Sample Stacks and CloudFormation Templates
CloudFormation can be useful for deploying infrastructure at AWS, including in the following areas:
Network: You can define a baseline template for developers to ensure that their VPC network deployment matches company policy.
Frontend infrastructure: You can deploy Internet gateways, associated route table entries, or load balancers into existing or new AWS network infrastructure.
Backend infrastructure: You can create database infrastructure, including primary and alternate database instances, subnet groups, and associated security groups.
Two-tier application: Using a two-tier CloudFormation script allows you to handle failures or disasters by enabling you to rebuild a complete application stack with required network and infrastructure components or to launch the application stack in another AWS region.
Windows Server Active Directory: You can deploy Active Directory on Windows Server 2008 R2 instances in a VPC.
Demo applications: You can define an application stack for demonstrations, thus allowing the sales team or end users to quickly create the entire environment.
AWS managed services: You can use CloudFormation templates to automate the setup of any AWS managed services. For example, you can enable and set up AWS Config or Inspector by using a CloudFormation template.
Creating an EC2 Instance
Example 3-4 provides a simple example that shows how to create an EC2 instance using a CloudFormation template. The template parameters are easily readable from top to bottom. Under Properties, the EC2 instance ID, subnet ID, and EC2 instance type must all be already present in the AWS region where the template is executed; if they are not, the deployment will fail. If there are issues in the CloudFormation script during deployment, CloudFormation rolls back and removes any infrastructure that the template created. The Ref statement is used in this template to attach the elastic IP (EIP) address to the defined EC2 instance that was deployed and referenced under the resources listed as EC2 Machine.
Example 3-4 CloudFormation Template for Creating an EC2 Instance
AWSTemplateFormatVersion: 2010-09-09
Description: EC2 Instance Template
"Resources": {
"EC2Machine": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId": "i-0ff407a7042afb0f0",
"NetworkInterfaces": [{
"DeviceIndex": "0",
"DeleteOnTermination": "true",
"SubnetId": "subnet-7c6dd651"
}]
"InstanceType": "t2.small"
}
}
},
"EIP": {
"Type": "AWS::EC2::EIP",
"Properties": {
"Domain": "VPC"
}
},
"VpcIPAssoc": {
"Type": "AWS::EC2::EIPAssociation",
"Properties": {
"InsanceId": {
"Ref": "EC2Machine"
},
"AllocationId": {
"Fn::GetAtt": ["EIP",
"AllocationId"]
}
}
}
Updating with Change Sets
Change sets allow you to preview how your existing AWS resources will be modified when a deployed CloudFormation resource stack needs to be updated (see Figure 3-15). You select an original CloudFormation template to edit and input the desired set of changes. CloudFormation then analyzes your requested changes against the existing CloudFormation stack and produces a change set that you can review and approve or cancel.
)
FIGURE 3-15 Using Change Sets with CloudFormation
You can create multiple change sets for various comparison purposes. Once a change set is created, reviewed, and approved, CloudFormation updates your current resource stack.
CloudFormation Stack Sets
A stack set allows you to create a single CloudFormation template to deploy, update, or delete AWS infrastructure across multiple AWS regions and AWS accounts. When a CloudFormation template will be deploying infrastructure across multiple accounts, as shown in Figure 3-16, and AWS regions, you must ensure that the AWS resources that the template references are available in each AWS account or region. For example, EC2 instances, EBS volumes, and key pairs are always created in a specific region. These region-specific resources must be copied to each AWS region where the CloudFormation template is executed. It is important to review global resources such as IAM roles and S3 buckets that are being created by the CloudFormation template to make sure there are no naming conflicts during creation, as global resources must be unique across all AWS regions.
)
FIGURE 3-16 A Stack Set with Two AWS Target Accounts
Once a stack set is updated, all instances of the stack that were created are updated as well. For example, if you have 10 AWS accounts across 3 AWS regions, 30 stack instances are updated when the primary stack set is executed. If a stack set is deleted, all corresponding stack sets are also deleted.
A stack set is first created in a single AWS account. Before additional stack instances can be created from the primary stack set, trust relationships using IAM roles must be created between the initial AWS administrator account and the desired target accounts.
For testing purposes, one example available in the AWS CloudFormation console is a sample stack set that allows you to enable AWS Config across selected AWS regions or accounts. Keep in mind that AWS Config allows you to control AWS account compliance by defining rules that monitor specific AWS resources to ensure that the desired level of compliance has been followed.
Third-Party Solutions
There are a number of third-party solutions, such as Chef, Puppet, Ansible, and TerraForm, for performing automated deployments of compute infrastructure. CloudFormation does not replace these third-party products but can be a useful tool for building automated solutions for your AWS infrastructure if you don’t use one of the third-party orchestration tools. AWS has a managed service called OpsWorks that comes in three flavors and might be useful to your deployments at AWS if your company currently uses Chef or Puppet:
AWS OpsWorks Stacks: This flavor allows you to manage applications and services that are hosted at AWS and on premises by running Chef recipes, bash, or PowerShell scripts.
AWS OpsWorks for Chef Automate: This flavor allows you to build a fully managed Chef Automate server that supports the latest versions of Chef server and Chef Automate, any community-based tools or cookbooks, and native Chef tools.
OpsWorks for Puppet Enterprise: This is a fully managed Puppet Enterprise environment that patches, updates, and backs up your existing Puppet environment and allows you to manage and administrate both Linux and Windows server nodes hosted on EC2 instances and on premises.
AWS Service Catalog
Using a CloudFormation template provides great power for creating, modifying, and updating AWS infrastructure. Creating AWS infrastructure always costs money. Say that you would like to control who gets to deploy specific CloudFormation templates. You can use Service Catalog to manage the distribution of CloudFormation templates as a product list to an AWS account ID, an AWS Organizations account, or an organizational unit contained within an AWS organization. Service Catalog is composed of portfolios, as shown in Figure 3-17, each of which is a collection of one or more products.
)
FIGURE 3-17 Portfolios in Service Catalog
When an approved product is selected, Service Catalog delivers a confirmation template to CloudFormation, which then executes the template and creates the product. Third-party products hosted in the AWS Marketplace are also supported by Service Catalog, and software appliances are bundled with a CloudFormation template.
Each IAM user in an AWS account can be granted access to a Server Catalog portfolio of multiple approved products. Because products are built using common confirmation templates, any AWS infrastructure components, including EC2 instances and databases hosted privately in a VPC, can be deployed. In addition, VPC endpoints using AWS PrivateLink allow access to the AWS Service Catalog service.
When you’re creating Service Catalog products, you can use constraints with IAM roles to limit the level of administrative access to the resources contained in the stack being deployed by the product itself. You can also assign server actions for rebooting, starting, or stopping deployed EC2 instances, as shown in Figure 3-18.
)
FIGURE 3-18 IAM Group Constraints Controlled by Service Catalog
In addition, you can add rules that control any parameter values that the end user enters. For example, you could mandate that specific subnets must be used for a stack deployment. You can also define rules that allow you to control which AWS account and region a product can launch.
If you list deployed products by version number, you can allow end users to select the latest versions of products so they can update older versions of currently deployed products. In this way, you can use CloudFormation and Service Catalog together to create a self-serve portal for developers consisting of portfolios and products.