Home > Articles

This chapter is from the book

Automating AWS Architecture

Many systems have been put in place over the years to help successfully manage and deploy complicated software applications on complicated hardware stacks.

If you look at the AWS cloud as an operating system hosted on the Internet, the one characteristic of AWS that stands above all others is the level of integrated automation used to deploy, manage, and recover AWS services. There is not a single AWS service offered that is not heavily automated for deployment and in its overall operation. When you order a virtual private network (VPN), it’s created and available in seconds. If you order an Elastic Compute Cloud (EC2) instance, either through the AWS Management Console or by using the AWS command-line interface (CLI) tools, it’s created and available in minutes. Automated processes provide the just-in-time response you want when you order cloud services.

AWS services are being changed, enhanced, and updated 24/7, with features and changes appearing every day. AWS as a whole is deployed and maintained using a combination of developer agility and automated processes; it is able to move quickly and easily with a partnership of developers, system operations, project managers, network engineers, and security professionals working together from the initial design stages, through the development process, to production and continual updates.

AWS wasn’t always so automated and regimented. In the early days, Amazon was a burgeoning online e-commerce bookseller. Not so very long ago, you would order a virtual machine from a cloud provider and wait several days for an email to arrive, telling you that your service was ready to go. As the Amazon e-commerce site became more popular, new problems appeared in the realm of scaling online resources to match customers’ needs. Over time, Amazon developed rules for all developers, mandating that each underlying service that supported the Amazon store must be accessible through a core set of shared application programming interfaces (APIs) that were shared with all developers; each service needed to be built and maintained on a common core of compute and storage resources.

Amazon built and continues to build its hosting environment using mandated internal processes, which can be described as a mixture of the following:

  • ITIL: Information Technology Infrastructure Library (ITIL) is a framework of best practices for delivering IT services.

  • Scrum: Scrum is a framework in which a development team works together to manage product development.

  • Agile: Agile is a software development cycle in which developers plan, design, develop, test, and evaluate as a team with open communications.

  • DevOps: DevOps is a continuation of the Agile framework that features full collaboration among the development and operations teams.

Many people at AWS are working together in an effective manner to make hundreds of changes to the AWS hardware and software environment every month. In addition, all AWS services are being monitored, scaled, rebuilt, and logged through completely automated processes. Amazon avoids using manual processes, and your long-term goal should be for your company to do the same.

As your experience with AWS grows, you’re going to want to start using automation to help run and manage your day-to-day AWS operations and to solve problems when they occur. There are numerous services available that don’t cost anything additional to use aside from the time it takes to become competent in using them. This might sound too good to be true, but most of Amazon’s automation services are indeed free to use; you are charged only for the AWS compute and storage resources that each service uses.

Automation services will always manage your resources more effectively than you can manually. At AWS, the automation of infrastructure is typically called infrastructure as code (IAC). When you create resources using the AWS Management Console, in the background, AWS uses automated processes running scripts to finish the creation and management of resources.

Regardless of how you define your own deployment or development process, there are a number of powerful tools in the AWS toolbox that can help you automate procedures:

  • CloudFormation: For stack-based architecture deployments

  • Service Catalog: To help secure CloudFormation templates

  • Elastic Beanstalk: For deploying applications and infrastructure together

Automating Infrastructure with CloudFormation

The second and third times you deploy EC2 instances using the AWS Management Console, you will probably not perform the steps the same way you did the first time. Even if you do manage to complete a manual task with the same steps, by the tenth installation your needs will have changed or better options will have become available. A manual process rarely stays the same over time. To make changes easier, you can automate even the simplest manual processes at AWS.

If you peek under the hood at any management service running at AWS, you’ll find the process command set driven by JavaScript Object Notation (JSON) scripts. At the GUI level, you use the Management Console to fill in the blanks; when you click Create, JSON scripts are executed in the background to carry out your requests. AWS’s extensive use of JSON is similar to Microsoft Azure’s extensive use of PowerShell; at AWS, JSON scripts are used internally for many tasks and processes; creating security policy with Identity and Access Management (IAM) and working with CloudFormation are two examples that you will commonly come across.

If you use Windows EC2 instances at AWS, you can also use PowerShell scripting. Both Microsoft Azure and AWS heavily rely on automation tools for everyday deployments. In the background, Azure relies heavily on PowerShell scripting and automation.

CloudFormation is an AWS-hosted orchestration engine that works with JSON templates to deploy AWS resources on demand or on predefined triggers (see Figure 3-13). AWS uses CloudFormation extensively, and so can you. More than 300,000 AWS customers use CloudFormation to manage deployment of just about everything, including all infrastructure stack deployments. A CloudFormation template declares the infrastructure stack to be created, and the CloudFormation engine automatically deploys and links the needed resources. You can add additional control variables to each CloudFormation template to manage and control the precise order of the installation of resources.


FIGURE 3-13 The CloudFormation Console

Consider this comparison of the manual AWS deployment process and the automated process that starts with CloudFormation:

  • Time spent: Over time, running manual processes at AWS becomes a big waste of time for the human carrying out the process. In the past, maintaining manual processes such as building computer systems and stacks provided job security; these days, it’s just not a prudent way to deploy production resources. For one thing, there are many steps in a manual process—which means there are many places the process can go wrong. Every CloudFormation deployment does take time, but it takes much less time than a manual process because each step in a CloudFormation script is essential. There are no wasted steps, and all steps are in the proper order. Over time, executing an automated process to build EC2 instances will save you hours or even weeks of time; the CloudFormation process runs in the background, allowing you to do something else. CloudFormation can also perform updates and deletions for existing AWS resources.

  • Security issues: Humans make mistakes, and manual changes can end up being huge security mistakes due to a lack of oversight. CloudFormation templates can be secured and controlled for usage by specific IAM users and groups. Templates also carry out the same steps every time they are executed, which prevents the fat-finger mistakes that humans make. The AWS Service Catalog service integrates with CloudFormation to mandate which users or accounts can use CloudFormation templates to build infrastructure stacks.

  • Documentation: It is difficult to document manual processes if they constantly change, and it can be difficult to find time to create documentation. CloudFormation templates are readable, and when you get used to the format, they are actually self-documenting. Again, there are no wasted steps in a CloudFormation script; what is described is exactly what is deployed. If mistakes are found in a CloudFormation script during deployment, all changes that have been carried out are reversed.

  • Repeatability: If you’re lucky, you can repeat your manual steps the same way every time. However, trying to do so is a waste of valuable time in the long run. With a CloudFormation script, you can deploy and redeploy the listed AWS resources in multiple environments, such as separate development, staging, and production environments. Every time a CloudFormation template is executed, it repeats the same steps.

  • Cost savings: CloudFormation automation carries out stack deployments and updates much faster than manual processes ever could. In addition, CloudFormation automation can be locked down using a companion service called AWS Systems Manager, discussed later in this chapter, to ensure that only specific IAM users and groups can access and execute specific CloudFormation deployment tasks.

CloudFormation Components

CloudFormation works with templates, stacks, and change sets. A CloudFormation template is an AWS resource blueprint that can create a complete application stack or a single stack component, such as a VPC network complete with multiple subnets, Internet gateways, and NAT services all automatically deployed and configured. You can create a template type called a change set to help visualize how proposed changes will affect AWS resources deployed by a CloudFormation template.

CloudFormation Templates

Each CloudFormation template is a text file that follows either JSON or YAML formatting standards. CloudFormation responds to files saved with JSON, YAML, and .txt extensions. Each template can deploy or update multiple AWS resources or a single resource, such as a VPC or an EC2 instance. Example 3-1 shows a CloudFormation template in JSON format, and Example 3-2 displays the same information in YAML format. It’s really a matter of personal preference which format you use. When creating CloudFormation templates, you might find YAML easier to read, which could be helpful in the long term. YAML seems more self-documenting because it’s easier to read.

Example 3-1 CloudFormation Template in JSON Format

    "AWSTemplateFormatVersion" : "2010-09-09",
    "Description" : "EC2 instance",
    "Resources" : {
    "EC2Instance" : {
    "Type" : "AWS::EC2::Instance",
    "Properties" : {
    "ImageId" : "ami-0ff8a91497e77f667",
    "InstanceType" : "t1.micro"

Example 3-2 CloudFormation Template in YAML Format

AWSTemplateFormatVersion: '2010-09-09'
Description: EC2 instance
Type: AWS::EC2::Instance
ImageId: ami-0ff8a91497e77f667

CloudFormation templates can have multiple sections, as shown in Example 3-3. However, the only mandatory section is Resources. As with any other template or script, the better the internal documentation, the more usable a CloudFormation template is—for the author as well as other individuals. It is highly recommended to use the Metadata section for comments to ensure that users understand the script.

Example 3-3 Valid Sections in a CloudFormation Template

"AWSTemplateFormatVersion": "version date",
"AWSTemplateFormatVersion": "2010-09-09"
<TemplateFormatVersion: Defines the current CF template version>
"Description": "Here are the additional details about this template
and what it does",
<Description: Describes the template: must always follow the version
"Metadata": {
   "Metadata" : {
   "Instances" :  {"Description : "Details about the instances"},
   "Databases" : {"Description : "Details about the databases"}
<Metadata: Additional information about the resources being deployed
by the template>
   "Parameters" : {
   "InstanceTypeParameter" : {
       "Type" : "String" ,
       "Default" : "t2.medium" , 
       "AllowedValues" : ["t2.medium", "m5.large", "m5.xlarge"],
       "Description" : "Enter t2.medium, m.5large, or m5.xlarge.
       Default is t2.medium."
<Parameters: Defines the AWS resource values allowed to be selected
and used by your template>
"Mappings" : {
     "RegionMap" : [
         "us-east-1           : { "HVM64 : "ami-0bb8a91508f77f868"},
         "us-west-1          : { "HVM64 : "ami-0cdb828fd58c52239"},
         "eu-west-1          : { "HVM64 : "ami-078bb4163c506cd88"},
         "us-southeast-1    : { "HVM64 : "ami-09999b978cc4dfc10"},
         "us-northeast-1    : { "HVM64 : "ami-06fd42961cd9f0d75"}
<Mappings: Defines conditional parameters defined by a "key"; in this
example, the AWS region and a set of AMI values to be used>
  "Conditions": {
  "CreateTestResources": {"Fn::Equals" : [{"Ref" : "EnvType"}, "test"]}
<Conditions: Defines dependencies between resources, such as the
order when resources are created, or where resources are created.
For example, "test" deploys the stack in the test environment >

CloudFormation Stacks

AWS has many sample CloudFormation templates that you can download from the online CloudFormation documentation, as shown in Figure 3-14, and deploy in many AWS regions. A CloudFormation stack can be as simple as a single VPC or as complex as a complete three-tier application stack, complete with the required network infrastructure and associated services.


FIGURE 3-14 AWS Sample Stacks and CloudFormation Templates

CloudFormation can be useful for deploying infrastructure at AWS, including in the following areas:

  • Network: You can define a baseline template for developers to ensure that their VPC network deployment matches company policy.

  • Frontend infrastructure: You can deploy Internet gateways, associated route table entries, or load balancers into existing or new AWS network infrastructure.

  • Backend infrastructure: You can create database infrastructure, including primary and alternate database instances, subnet groups, and associated security groups.

  • Two-tier application: Using a two-tier CloudFormation script allows you to handle failures or disasters by enabling you to rebuild a complete application stack with required network and infrastructure components or to launch the application stack in another AWS region.

  • Windows Server Active Directory: You can deploy Active Directory on Windows Server 2008 R2 instances in a VPC.

  • Demo applications: You can define an application stack for demonstrations, thus allowing the sales team or end users to quickly create the entire environment.

  • AWS managed services: You can use CloudFormation templates to automate the setup of any AWS managed services. For example, you can enable and set up AWS Config or Inspector by using a CloudFormation template.

Creating an EC2 Instance

Example 3-4 provides a simple example that shows how to create an EC2 instance using a CloudFormation template. The template parameters are easily readable from top to bottom. Under Properties, the EC2 instance ID, subnet ID, and EC2 instance type must all be already present in the AWS region where the template is executed; if they are not, the deployment will fail. If there are issues in the CloudFormation script during deployment, CloudFormation rolls back and removes any infrastructure that the template created. The Ref statement is used in this template to attach the elastic IP (EIP) address to the defined EC2 instance that was deployed and referenced under the resources listed as EC2 Machine.

Example 3-4 CloudFormation Template for Creating an EC2 Instance

AWSTemplateFormatVersion: 2010-09-09
Description: EC2 Instance Template
"Resources": {
 "EC2Machine": {
 "Type": "AWS::EC2::Instance",
 "Properties": {
   "ImageId": "i-0ff407a7042afb0f0",
   "NetworkInterfaces": [{
   "DeviceIndex": "0",
   "DeleteOnTermination": "true",
   "SubnetId": "subnet-7c6dd651"
   "InstanceType": "t2.small"
"EIP": {
  "Type": "AWS::EC2::EIP",
  "Properties": {
  "Domain": "VPC"
"VpcIPAssoc": {
"Type": "AWS::EC2::EIPAssociation",
  "Properties": {
  "InsanceId":  {
  "Ref": "EC2Machine"
  "AllocationId": {
  "Fn::GetAtt": ["EIP",
Updating with Change Sets

Change sets allow you to preview how your existing AWS resources will be modified when a deployed CloudFormation resource stack needs to be updated (see Figure 3-15). You select an original CloudFormation template to edit and input the desired set of changes. CloudFormation then analyzes your requested changes against the existing CloudFormation stack and produces a change set that you can review and approve or cancel.


FIGURE 3-15 Using Change Sets with CloudFormation

You can create multiple change sets for various comparison purposes. Once a change set is created, reviewed, and approved, CloudFormation updates your current resource stack.

CloudFormation Stack Sets

A stack set allows you to create a single CloudFormation template to deploy, update, or delete AWS infrastructure across multiple AWS regions and AWS accounts. When a CloudFormation template will be deploying infrastructure across multiple accounts, as shown in Figure 3-16, and AWS regions, you must ensure that the AWS resources that the template references are available in each AWS account or region. For example, EC2 instances, EBS volumes, and key pairs are always created in a specific region. These region-specific resources must be copied to each AWS region where the CloudFormation template is executed. It is important to review global resources such as IAM roles and S3 buckets that are being created by the CloudFormation template to make sure there are no naming conflicts during creation, as global resources must be unique across all AWS regions.


FIGURE 3-16 A Stack Set with Two AWS Target Accounts

Once a stack set is updated, all instances of the stack that were created are updated as well. For example, if you have 10 AWS accounts across 3 AWS regions, 30 stack instances are updated when the primary stack set is executed. If a stack set is deleted, all corresponding stack sets are also deleted.

A stack set is first created in a single AWS account. Before additional stack instances can be created from the primary stack set, trust relationships using IAM roles must be created between the initial AWS administrator account and the desired target accounts.

For testing purposes, one example available in the AWS CloudFormation console is a sample stack set that allows you to enable AWS Config across selected AWS regions or accounts. Keep in mind that AWS Config allows you to control AWS account compliance by defining rules that monitor specific AWS resources to ensure that the desired level of compliance has been followed.

Third-Party Solutions

There are a number of third-party solutions, such as Chef, Puppet, Ansible, and TerraForm, for performing automated deployments of compute infrastructure. CloudFormation does not replace these third-party products but can be a useful tool for building automated solutions for your AWS infrastructure if you don’t use one of the third-party orchestration tools. AWS has a managed service called OpsWorks that comes in three flavors and might be useful to your deployments at AWS if your company currently uses Chef or Puppet:

  • AWS OpsWorks Stacks: This flavor allows you to manage applications and services that are hosted at AWS and on premises by running Chef recipes, bash, or PowerShell scripts.

  • AWS OpsWorks for Chef Automate: This flavor allows you to build a fully managed Chef Automate server that supports the latest versions of Chef server and Chef Automate, any community-based tools or cookbooks, and native Chef tools.

  • OpsWorks for Puppet Enterprise: This is a fully managed Puppet Enterprise environment that patches, updates, and backs up your existing Puppet environment and allows you to manage and administrate both Linux and Windows server nodes hosted on EC2 instances and on premises.

AWS Service Catalog

Using a CloudFormation template provides great power for creating, modifying, and updating AWS infrastructure. Creating AWS infrastructure always costs money. Say that you would like to control who gets to deploy specific CloudFormation templates. You can use Service Catalog to manage the distribution of CloudFormation templates as a product list to an AWS account ID, an AWS Organizations account, or an organizational unit contained within an AWS organization. Service Catalog is composed of portfolios, as shown in Figure 3-17, each of which is a collection of one or more products.


FIGURE 3-17 Portfolios in Service Catalog

When an approved product is selected, Service Catalog delivers a confirmation template to CloudFormation, which then executes the template and creates the product. Third-party products hosted in the AWS Marketplace are also supported by Service Catalog, and software appliances are bundled with a CloudFormation template.

Each IAM user in an AWS account can be granted access to a Server Catalog portfolio of multiple approved products. Because products are built using common confirmation templates, any AWS infrastructure components, including EC2 instances and databases hosted privately in a VPC, can be deployed. In addition, VPC endpoints using AWS PrivateLink allow access to the AWS Service Catalog service.

When you’re creating Service Catalog products, you can use constraints with IAM roles to limit the level of administrative access to the resources contained in the stack being deployed by the product itself. You can also assign server actions for rebooting, starting, or stopping deployed EC2 instances, as shown in Figure 3-18.


FIGURE 3-18 IAM Group Constraints Controlled by Service Catalog

In addition, you can add rules that control any parameter values that the end user enters. For example, you could mandate that specific subnets must be used for a stack deployment. You can also define rules that allow you to control which AWS account and region a product can launch.

If you list deployed products by version number, you can allow end users to select the latest versions of products so they can update older versions of currently deployed products. In this way, you can use CloudFormation and Service Catalog together to create a self-serve portal for developers consisting of portfolios and products.

Pearson IT Certification Promotional Mailings & Special Offers

I would like to receive exclusive offers and hear about products from Pearson IT Certification and its family of brands. I can unsubscribe at any time.


Pearson Education, Inc., 221 River Street, Hoboken, New Jersey 07030, (Pearson) presents this site to provide information about Pearson IT Certification products and services that can be purchased through this site.

This privacy notice provides an overview of our commitment to privacy and describes how we collect, protect, use and share personal information collected through this site. Please note that other Pearson websites and online products and services have their own separate privacy policies.

Collection and Use of Information

To conduct business and deliver products and services, Pearson collects and uses personal information in several ways in connection with this site, including:

Questions and Inquiries

For inquiries and questions, we collect the inquiry or question, together with name, contact details (email address, phone number and mailing address) and any other additional information voluntarily submitted to us through a Contact Us form or an email. We use this information to address the inquiry and respond to the question.

Online Store

For orders and purchases placed through our online store on this site, we collect order details, name, institution name and address (if applicable), email address, phone number, shipping and billing addresses, credit/debit card information, shipping options and any instructions. We use this information to complete transactions, fulfill orders, communicate with individuals placing orders or visiting the online store, and for related purposes.


Pearson may offer opportunities to provide feedback or participate in surveys, including surveys evaluating Pearson products, services or sites. Participation is voluntary. Pearson collects information requested in the survey questions and uses the information to evaluate, support, maintain and improve products, services or sites; develop new products and services; conduct educational research; and for other purposes specified in the survey.

Contests and Drawings

Occasionally, we may sponsor a contest or drawing. Participation is optional. Pearson collects name, contact information and other information specified on the entry form for the contest or drawing to conduct the contest or drawing. Pearson may collect additional personal information from the winners of a contest or drawing in order to award the prize and for tax reporting purposes, as required by law.


If you have elected to receive email newsletters or promotional mailings and special offers but want to unsubscribe, simply email information@informit.com.

Service Announcements

On rare occasions it is necessary to send out a strictly service related announcement. For instance, if our service is temporarily suspended for maintenance we might send users an email. Generally, users may not opt-out of these communications, though they can deactivate their account information. However, these communications are not promotional in nature.

Customer Service

We communicate with users on a regular basis to provide requested services and in regard to issues relating to their account we reply via email or phone in accordance with the users' wishes when a user submits their information through our Contact Us form.

Other Collection and Use of Information

Application and System Logs

Pearson automatically collects log data to help ensure the delivery, availability and security of this site. Log data may include technical information about how a user or visitor connected to this site, such as browser type, type of computer/device, operating system, internet service provider and IP address. We use this information for support purposes and to monitor the health of the site, identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents and appropriately scale computing resources.

Web Analytics

Pearson may use third party web trend analytical services, including Google Analytics, to collect visitor information, such as IP addresses, browser types, referring pages, pages visited and time spent on a particular site. While these analytical services collect and report information on an anonymous basis, they may use cookies to gather web trend information. The information gathered may enable Pearson (but not the third party web trend services) to link information with application and system log data. Pearson uses this information for system administration and to identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents, appropriately scale computing resources and otherwise support and deliver this site and its services.

Cookies and Related Technologies

This site uses cookies and similar technologies to personalize content, measure traffic patterns, control security, track use and access of information on this site, and provide interest-based messages and advertising. Users can manage and block the use of cookies through their browser. Disabling or blocking certain cookies may limit the functionality of this site.

Do Not Track

This site currently does not respond to Do Not Track signals.


Pearson uses appropriate physical, administrative and technical security measures to protect personal information from unauthorized access, use and disclosure.


This site is not directed to children under the age of 13.


Pearson may send or direct marketing communications to users, provided that

  • Pearson will not use personal information collected or processed as a K-12 school service provider for the purpose of directed or targeted advertising.
  • Such marketing is consistent with applicable law and Pearson's legal obligations.
  • Pearson will not knowingly direct or send marketing communications to an individual who has expressed a preference not to receive marketing.
  • Where required by applicable law, express or implied consent to marketing exists and has not been withdrawn.

Pearson may provide personal information to a third party service provider on a restricted basis to provide marketing solely on behalf of Pearson or an affiliate or customer for whom Pearson is a service provider. Marketing preferences may be changed at any time.

Correcting/Updating Personal Information

If a user's personally identifiable information changes (such as your postal address or email address), we provide a way to correct or update that user's personal data provided to us. This can be done on the Account page. If a user no longer desires our service and desires to delete his or her account, please contact us at customer-service@informit.com and we will process the deletion of a user's account.


Users can always make an informed choice as to whether they should proceed with certain services offered by Adobe Press. If you choose to remove yourself from our mailing list(s) simply visit the following page and uncheck any communication you no longer want to receive: www.pearsonitcertification.com/u.aspx.

Sale of Personal Information

Pearson does not rent or sell personal information in exchange for any payment of money.

While Pearson does not sell personal information, as defined in Nevada law, Nevada residents may email a request for no sale of their personal information to NevadaDesignatedRequest@pearson.com.

Supplemental Privacy Statement for California Residents

California residents should read our Supplemental privacy statement for California residents in conjunction with this Privacy Notice. The Supplemental privacy statement for California residents explains Pearson's commitment to comply with California law and applies to personal information of California residents collected in connection with this site and the Services.

Sharing and Disclosure

Pearson may disclose personal information, as follows:

  • As required by law.
  • With the consent of the individual (or their parent, if the individual is a minor)
  • In response to a subpoena, court order or legal process, to the extent permitted or required by law
  • To protect the security and safety of individuals, data, assets and systems, consistent with applicable law
  • In connection the sale, joint venture or other transfer of some or all of its company or assets, subject to the provisions of this Privacy Notice
  • To investigate or address actual or suspected fraud or other illegal activities
  • To exercise its legal rights, including enforcement of the Terms of Use for this site or another contract
  • To affiliated Pearson companies and other companies and organizations who perform work for Pearson and are obligated to protect the privacy of personal information consistent with this Privacy Notice
  • To a school, organization, company or government agency, where Pearson collects or processes the personal information in a school setting or on behalf of such organization, company or government agency.


This web site contains links to other sites. Please be aware that we are not responsible for the privacy practices of such other sites. We encourage our users to be aware when they leave our site and to read the privacy statements of each and every web site that collects Personal Information. This privacy statement applies solely to information collected by this web site.

Requests and Contact

Please contact us about this Privacy Notice or if you have any requests or questions relating to the privacy of your personal information.

Changes to this Privacy Notice

We may revise this Privacy Notice through an updated posting. We will identify the effective date of the revision in the posting. Often, updates are made to provide greater clarity or to comply with changes in regulatory requirements. If the updates involve material changes to the collection, protection, use or disclosure of Personal Information, Pearson will provide notice of the change through a conspicuous notice on this site or other appropriate way. Continued use of the site after the effective date of a posted revision evidences acceptance. Please contact us if you have questions or concerns about the Privacy Notice or any objection to any revisions.

Last Update: November 17, 2020