CloudFormation in Practice: Change Sets and Drift

A CloudFormation template that deploys cleanly the first time is the easy case. The hard case is the production stack that has been running for two years, accumulated drift, and now needs an update without breaking anything.

This chapter covers the operational tools that make CloudFormation safe at scale: change sets, drift detection, stack policies, and nested stacks.

Change Sets: Preview Before You Apply

A change set is a dry run. You submit a new template, and CloudFormation tells you exactly what it would do — what it would create, modify, or delete — before you give it the green light to actually do it.

Three flavors of change exist, and they behave very differently:

• Update with no interruption — CloudFormation modifies the resource in place. Safe;
• Update with some interruption — the resource briefly goes offline. Acceptable for most workloads;
• Replacement — CloudFormation creates a new resource, then deletes the old one. The resource gets a new ID. Sometimes catastrophic. The classic horror story: renaming an RDS instance's DBInstanceIdentifier triggers a replacement, which spins up a fresh empty database and deletes the original — with the production data inside it. A change set would have shown "Replacement: True" in big red letters before that happened.

The rule: always create a change set for production updates. Always read it. Never trust the diff in your editor.

Drift Detection: Catching ClickOps

Drift is the gap between what the template says and what actually exists in AWS. Someone went into the console and changed a security group rule. The template does not know. CloudFormation will happily overwrite the manual change on the next deploy — or worse, fail mysteriously.

Drift detection scans your stack and reports anything that no longer matches the template. Run it on a schedule. Investigate any drift before the next deploy.

Common drift causes:

• Console edits during incidents ("we will put it back in the template later");
• Auto-scaling groups changing tag values dynamically;
• Lambda functions deployed by a separate CI tool, not by CloudFormation;
• Manual IAM policy edits to fix a permissions error in a hurry.

Stack Policies: Protecting Critical Resources

A stack policy is a JSON document attached to a stack that restricts which resources can be updated. The classic use: protect a production RDS database from accidental replacement.

{
  "Statement": [
    {
      "Effect": "Deny",
      "Action": "Update:Replace",
      "Principal": "*",
      "Resource": "LogicalResourceId/ProductionDatabase"
    }
  ]
}

With this in place, any change set that would replace the database fails — instead of silently proceeding.

Nested Stacks: Reusable Building Blocks

Once a template hits a few hundred lines, you want to split it. Nested stacks let you reference another CloudFormation template as a resource:

NetworkStack:
  Type: AWS::CloudFormation::Stack
  Properties:
    TemplateURL: https://s3.amazonaws.com/templates/network.yaml

A common pattern: one root stack that nests a VPC stack, a database stack, and an application stack. Each can be developed and tested independently, then composed.

StackSets: Across Accounts and Regions

For multi-account, multi-region work, StackSets deploy the same template across many targets at once. Useful for:

• Enforcing a baseline like a logging bucket and CloudTrail in every account;
• Deploying a network template to every region your company operates in;
• Rolling out the same IAM role to every account in an AWS Organization.

The Day-to-Day Workflow

For a production stack, the safe sequence is:

• Edit the template in a branch;
• Open a pull request, get a review;
• Merge and let CI create a change set against production;
• A human reads the change set output;
• A human approves;
• CI executes the change set;
• Drift detection runs nightly to catch anything that slipped through. This sounds heavy until the day a change set says "Replacement: True" on your production database. Then it sounds cheap.