Terraform Networking Trifecta

Intro

Initially I’d planned to blog more than just about Terraform but it hasn’t turned out that way due to lack of time.

But that’s OK because I really do enjoy creating and writing about networking topics with Terraform in the cloud.

Content generation is hard so hats off to those who are consistently generating it.

This project is for all the Terraform headz building networking in the cloud!

It’s building on my previous Tiered VPC idea but by adding a couple more abstractions we can build flexible networking between multiple AWS VPCs with minimal code.

Goal

Using Terraform (v1.0.4+) and AWS Provider (v3.53.0+) route between 3 VPCs with different IPv4 CIDR ranges (RFC 1918) using a Transit Gateway.

  • App VPC Tier: 10.0.0.0/20 (Class A Private Internet)
  • CICD VPC Tier: 172.16.0.0/20 (Class B Private Internet)
  • General VPC Tier: 192.168.0.0/20 (Class C Private Internet)

link

Modules:

Main:

Inspiration Begets Motivation

Simple yet boring cloud diagrams are cool and all but I need inspiration to get motivated so peep the visuals to spice up the concept.

https://twitter.com/MAKIO135/status/1378469836305666055 https://twitter.com/MAKIO135/status/1380634991818911746 https://twitter.com/MAKIO135/status/1379931304863657984 https://twitter.com/MAKIO135/status/1404543066724253699 https://twitter.com/MAKIO135/status/1368340867803660289

Big shout out Lionel Radisson!

Tiered VPC-NG

The new Tiered VPC-NG module is based off the old Tiered VPC prototype.

NG (next generation) is a finalized design based off the learnings of its predecessor.

It was created separately in an effort not to lose the context and frame of mind in the original Synthesizing Tiered VPC in Terraform article.

Combining multiple ideas forces them to evolve into something new.

Most importanlty, calling my creations next gen is fun because it really doesn’t matter!

Baseline Tiered VPC-NG features (same as prototype):

  • Create VPC tiers

    • It’s much easier to think about tiers ephemerally when scaling out VPCs because we can narrow down the context (ie app, db, general) and maximize the use of smaller network size when needed.
    • You can still have 3 tiered networking (ie LBs, DBs, App) internally to the VPC depending on the type of tier you’re building.
  • VPC resources are IPv4 only.

    • No IPv6 configuration for now.
  • Creates structured VPC resource naming and tagging.

    • Ephemeral Naming
    • <env_prefix>-<region|az>-<tier_name>-<public|private>-<pet_name|misc_label>
  • Requires a minimum of at least one public subnet per AZ.

  • Can add and remove subnets and/or AZs at any time*.

  • Internal and external VPC routing is automatic.

What’s new in NG?

  • An Intra VPC Security Group is created by default.

    • This will be for adding security group rules that are inbound only for access across VPCs.
  • No more nulling out private subnets ie private = null.

    • Just populate the list to create them ie private = [] just like the public subnets.
  • NAT Gateways are no longer built by default when private subnets exist in an AZ.

    • This allows for routable private subnets that have no outbound internet traffic unless enable_natgw = true, in which case the NAT Gateway will be built and private route tables updated.
    • Now you can just “flip the switch” to give all private subnets in an AZ access to the internet.
    • NAT Gateways are created with an EIP per AZ when enabled.
      • This is why we need at least one public subnet per AZ to create them in.

Example tier:

tier = {
  azs = {
    a = {
      # internally routable private subnets but with no routing for
      # outbound internet traffic
      private = ["192.168.0.0/24", "192.168.1.0/24"]
      public  = ["192.168.3.0/28"]
    }
    b = {
      # internally routable private subnet with routing for
      # outbound internet traffic
      enable_natgw = true
      private = ["192.168.5.0/24", "192.168.6.0/24"]
      public  = ["192.168.8.0/28"]
    }
    c = {
      # no private subnets exist so a NAT Gateway won't be created until
      # the private subnet list is poplulated.
      enable_natgw = true
      private = []
      public  = ["192.168.13.0/28"]
    }
  }
  name    = "general"
  network = "192.168.0.0/20"
}

I’ve also settled on these decomposed structures for outputs.tf that other modules will call/consume.

Here is an example of building out 3 VPCs each with a different /20 network, almost fully subnetted.

# snippet
locals {
  vpc_tiers = [
    {
      azs = {
        a = {
          private = ["10.0.0.0/24", "10.0.1.0/24", "10.0.2.0/24"]
          public  = ["10.0.3.0/24", "10.0.4.0/24"]
        }
        b = {
          enable_natgw = true
          private = ["10.0.5.0/24", "10.0.6.0/24", "10.0.7.0/24"]
          public  = ["10.0.8.0/24", "10.0.9.0/24"]
        }
        c = {
          private = ["10.0.10.0/24", "10.0.11.0/24", "10.0.12.0/24"]
          public  = ["10.0.13.0/24", "10.0.14.0/24"]
        }
      }
      name    = "app"
      network = "10.0.0.0/20"
    },
    {
      azs = {
        a = {
          private = ["172.16.0.0/24", "172.16.1.0/24", "172.16.2.0/24"]
          public  = ["172.16.3.0/28", "172.16.4.0/28"]
        }
        b = {
          enable_natgw = true
          private = ["172.16.5.0/24", "172.16.6.0/24", "172.16.7.0/24"]
          public  = ["172.16.8.0/28", "172.16.9.0/28"]
        }
        c = {
          private = ["172.16.10.0/24", "172.16.11.0/24", "172.16.12.0/24"]
          public  = ["172.16.13.0/28", "172.16.14.0/28"]
        }
      }
      name    = "cicd"
      network = "172.16.0.0/20"
    },
    {
      azs = {
        a = {
          enable_natgw = true
          private = ["192.168.0.0/24", "192.168.1.0/24", "192.168.2.0/24"]
          public  = ["192.168.13.0/28"]
        }
        b = {
          private = ["192.168.5.0/24", "192.168.6.0/24", "192.168.7.0/24"]
          public  = ["192.168.8.0/28"]
        }
        c = {
          private = ["192.168.10.0/24", "192.168.11.0/24", "192.168.12.0/24"]
          public  = ["192.168.13.0/28"]
        }
      }
      name    = "general"
      network = "192.168.0.0/20"
    }
  ]
}

module "vpcs" {
  source = "git@github.com:JudeQuintana/terraform-modules.git//networking/tiered_vpc_ng?ref=v1.2.0"

  for_each = { for t in local.vpc_tiers : t.name => t }

  env_prefix       = var.env_prefix
  region_az_labels = var.region_az_labels
  tier             = each.value
}

Security Group Rules

The Intra VPC Security Group Rule module will create an inbound only rule for each VPC that will point to all other VPC networks except the VPC itself.

This will allow ssh and ping protocols accross Tiered VPCs so communication will just work once we enable routing to each other with Transit Gateway.

# snipppet
locals {
  intra_vpcs_security_group_rules = [
    {
      label     = "ssh"
      from_port = 22
      to_port   = 22
      protocol  = "tcp"
    },
    {
      label     = "ping"
      from_port = 8
      to_port   = 0
      protocol  = "icmp"
    }
  ]
}

module "intra_vpcs_security_group_rules" {
  source = "git@github.com:JudeQuintana/terraform-modules.git//networking/intra_vpc_security_group_rule_for_tiered_vpc_ng?ref=v1.2.0"

  for_each = { for r in local.intra_vpcs_security_group_rules : r.label => r }

  env_prefix = var.env_prefix
  vpcs       = module.vpcs
  rule       = each.value
}

It’s important that the Secruity Group rules are created in a seperate module from the routing.

Transit Gateway (TGW)

This Transit Gateway Centralized Router module will attach all AZs in each Tiered VPC to the TGW, think router on a stick!

All attachments will be associated and routes propagated to one TGW Route Table.

Each Tiered VPC will have all their route tables updated in each VPC with a route to all other VPC networks via the TGW.

# snippet
module "tgw_centralized_router" {
  source = "git@github.com:JudeQuintana/terraform-modules.git//networking/transit_gateway_centralized_router_for_tiered_vpc_ng?ref=v1.2.0"

  env_prefix       = var.env_prefix
  region_az_labels = var.region_az_labels
  vpcs             = module.vpcs
}

I didn’t add any TGW peering, cross-account/cross-region sharing, or specific TGW routes (ie aws_ec2_transit_gateway_route) at this time.

I’m still learning how to put TGW sharing together for the next iteration.

Caveats

The modules build resources that will cost some money but should be minimal for the demo. (ie NATGW, EIP, TGW)

You wont be able to see the generated pet names for subnet naming on plan, only on apply.

  • Even though you can delete subnets in a VPC, remember that the NAT Gateways get created in the first public subnet in the list for the AZ.

When modifying an AZ or VPCs in an existing configuration with A TGW Centralized rouer:

  • Adding
    • The VPCs must be applied first.
    • Then apply Intra Security Groups Rules and TGW Centralzed Router.
  • Removing
    • The first public subnet in the AZ being removed must be manually removed (modified) from the TGW VPC attachments first before applying to the VPC, SG Rules, and TGW.
    • Otherwise, Terraform wants to remove the attachment from the TGW after the subnet is deleted but the subnet can’t be deleted until it’s attachment is removed from the TGW (strange circular dependency that I haven’t figured out).
    • Full teardown (destroy) works fine.

Trifecta Demo Time

This will be a demo of the following:

  • Configure us-west-2a AZ in app VPC - 10.0.0.0/20
    • Launch app-public instance in public subnet.
  • Configure us-west-2b AZ with NATGW in cicd VPC - 172.16.0.0/20
    • Launch cicd-private instance in private subnet.
  • Configure us-west-2c AZ in general VPC - 192.168.0.0/20
    • Launch general-private instance in private subnet.
  • Configure security groups for access across VPCs.
    • Allow ssh and ping.
  • Configure routing between all public and private subnets accross VPCs via TGW.
  • Verify connectivity with t2.micro EC2 instances.
  • Minimal assembly required.

Pre-requisites:

  • git
  • curl
  • Terraform 1.0.4+
  • Configured AWS credentials
    • An AWS EC2 Key Pair should already exist in the us-west-2 region and the private key should have user read only permissions.
      • private key saved as ~/.ssh/my-ec2-key.pem on local machine.
      • must be user read only permssions chmod 400 ~/.ssh/my-ec2-key.pem

Assemble the Trifecta by cloning the Networking Trifecta Demo repo.

$ git clone git@github.com:JudeQuintana/terraform-main.git
$ cd networking_trifecta_demo

Update the key_name in variables.tf with the EC2 key pair name you’re using for the us-west-2 region (see pre-requisites above).

# snippet
variable "base_ec2_instance_attributes" {
  ...
  default = {
    key_name      = "my-ec2-key"            # EC2 key pair name to use when launching an instance
    ami           = "ami-0518bb0e75d3619ca" # AWS Linux 2 us-west-2
    instance_type = "t2.micro"
  }
}

The VPCs must be applied first:

$ terraform init
$ terraform apply -target module.vpcs

Now, apply secrity groups to allow ssh and ping across VPCs, launch instances in each enabled AZ for all VPCs, and route between VPCs via TGW:

$ terraform apply -target module.intra_vpc_security_group_rules -target aws_instance.instances -target module.tgw_centralized_router

After the TGW apply is complete, it will take 1-2 minutes for the TGW routing to fully propagate.

Verify Connectivity Between VPCs

$ cd ../scripts/
$ chmod u+x get_instance_info.sh
$ ./get_instance_info.sh

Example output:

# aws_instance.instances["cicd-private"]
    private_ip                           = "172.16.5.11"

# aws_instance.instances["general-private"]
    private_ip                           = "192.168.10.8"

# aws_instance.instances["app-public"]
    private_ip                           = "10.0.3.200"
    public_ip                            = "54.187.241.115"

# module.vpcs["app"].aws_vpc.this
    default_security_group_id        = "sg-id-1234"

# My Public IP
XX.XXX.XXX.XX

Using the output above, add an inbound ssh rule from “My Public IP” to the default security group id of the App VPC.

This can be done manually in the AWS console or with the aws cli command below.

$ aws ec2 authorize-security-group-ingress --group-id sg-id-1234 --protocol tcp --port 22 --cidr My.Public.IP.Here/32

Next, ssh to the app-public instance public IP (ie 54.187.241.115) using the EC2 key pair private key.

Then, ssh to the private_ip of the general-private instance, then cicd-private, then back to app-public.

$ ssh -i ~/.ssh/my-ec2-key.pem ec2-user@54.187.241.115

[ec2-user@app-public ~]$ ping google.com # works! via igw
[ec2-user@app-public ~]$ ping 192.168.10.8 # works! via tgw
[ec2-user@app-public ~]$ ssh 192.168.10.8

[ec2-user@general-private ~]$ ping google.com # doesn't work! no natgw
[ec2-user@general-private ~]$ ping 172.16.5.11 # works! via tgw
[ec2-user@general-private ~]$ ssh 172.16.5.11

[ec2-user@cicd-private ~]$ ping google.com # works! via natgw
[ec2-user@cicd-private ~]$ ping 10.0.3.200 # works! via tgw
[ec2-user@cicd-private ~]$ ssh 10.0.3.200

[ec2-user@app-public ~]$

🔻 Trifecta Complete!!!

Clean Up

$ terraform destroy -target module.vpcs -target module.intra_vpc_security_group_rules -target aws_instance.instances -target module.tgw_centralized_router

Final Thoughts

This article endeding up being way more work than I thought was needed so thanks for taking the time to read about my random cloud networking ideas, stay up!

~jq1

Feedback

What did you think about this post? jude@jq1.io