jq1.io
Author Image

Ideas as Code

© 2023 Jude Quintana
Built with Hugo

Terraform Networking Trifecta

Jul 22, 2021 in TERRAFORM
aws vpc tgw security-group-rules routing subnetting hub-spoke-topology cloud-networking
10 min read

Intro

Initially I’d planned to blog more than just about Terraform but it hasn’t turned out that way due to lack of time.

But that’s OK because I really do enjoy creating and writing about networking topics with Terraform in the cloud.

Content generation is hard so hats off to those who are consistently generating it.

This project is for all the Terraform headz building networks in the cloud!

It’s building on my previous Tiered VPC idea but by adding a couple more abstractions we can build flexible networking between multiple AWS VPCs with minimal code.

Goal

Using Terraform (v1.3+) and AWS Provider (v4.20+) to route between 3 VPCs with different IPv4 CIDR ranges (RFC 1918) using a Transit Gateway.

  • App VPC Tier: 10.0.0.0/20 (Class A Private Internet)
  • CICD VPC Tier: 172.16.0.0/20 (Class B Private Internet)
  • General VPC Tier: 192.168.0.0/20 (Class C Private Internet)

Example VPC-NG architecture but with auto named subnets: vpc-ng

The resulting architecture is a hub spoke topology (zoom out): tnt

Modules:

Main:

Inspiration Begets Motivation

Simple yet boring cloud diagrams are cool and all but I need inspiration to get motivated so peep the visuals to spice up the concept.

https://twitter.com/MAKIO135/status/1378469836305666055 https://twitter.com/MAKIO135/status/1380634991818911746 https://twitter.com/MAKIO135/status/1379931304863657984 https://twitter.com/MAKIO135/status/1404543066724253699 https://twitter.com/MAKIO135/status/1368340867803660289

Big shout out Lionel Radisson!

TNT Architecture

Tiered VPC-NG

The new Tiered VPC-NG module is based off the old Tiered VPC prototype.

NG is supposed to be a finalized design based off the learnings of it’s predecessor but it’s just cool to call my projects next gen!

It’s fun to combine multiple ideas and watch them to evolve into something new.

The Tiered VPC-NG module was created separately in an effort not to lose the context and frame of mind in the original Synthesizing Tiered VPC in Terraform article.

Baseline Tiered VPC-NG features (same as prototype):

  • Create VPC tiers

    • It’s much easier to think about tiers ephemerally when scaling out VPCs because we can narrow down the context (ie app, db, general) and maximize the use of smaller network size when needed.
    • You can still have n-tiered networking (ie lbs, asg, dbs for an app) internally to the VPC.

  • VPC resources are IPv4 only.

    • No IPv6 configuration for now.

  • Creates structured VPC resource naming and tagging.

    • Auto Naming
    • <env_prefix>-tiered-vpc-<region|az>-<tier_name>-<public|private>-<pet_name|misc_label>

  • Requires a minimum of at least one public subnet per AZ.

    • Trade-offs
      • Eats up a public subnet even if we are only going to use private subnets.
      • Allows us to simply declare enable_natgw = true in the AZ block of the VPC and build the NATGW in the first public subnet in the AZ.
      • Allows Centralized Router to easily filter and select the first public subnet in each enabled AZ to build the vpc attachments because it will always exist.

  • Can add and remove subnets and/or AZs at any time*.

  • Internal and external VPC routing is automatic.

What’s new in NG?

  • An Intra VPC Security Group is created by default.

    • This will be for adding security group rules that are inbound only for access across VPCs.

  • No more nulling out private subnets ie private = null.

    • Just populate the list to create them ie private = [] just like the public subnets.

  • NAT Gateways are no longer built by default when private subnets exist in an AZ.

    • This allows private subnets that have no outbound internet traffic unless enable_natgw = true, in which case the NAT Gateway will be built in a public subnet and private route tables updated.
    • Now you can just “flip the switch” to give all private subnets in an AZ access to the internet.
    • NAT Gateways are created with an EIP per AZ when enabled.
      • This is why we need at least one public subnet per AZ to create them in.

Example tier:

tier = {
  azs = {
    a = {
      # internally routable private subnets but with no routing for
      # outbound internet traffic
      private = ["192.168.0.0/24", "192.168.1.0/24"]
      public  = ["192.168.3.0/28"]
    }
    b = {
      # internally routable private subnet with routing for
      # outbound internet traffic
      enable_natgw = true
      private = ["192.168.5.0/24", "192.168.6.0/24"]
      public  = ["192.168.8.0/28"]
    }
    c = {
      # no private subnets exist but a NAT Gateway will be created
      enable_natgw = true
      private = []
      public  = ["192.168.13.0/28"]
    }
  }
  name    = "general"
  network = "192.168.0.0/20"
}

I’ve also settled on these decomposed structures for outputs.tf that other modules will call/consume.

Here is an example of building out 3 VPCs each with a different /20 network, almost fully subnetted.

# snippet
locals {
  vpc_tiers = [
    {
      azs = {
        a = {
          private = ["10.0.0.0/24", "10.0.1.0/24", "10.0.2.0/24"]
          public  = ["10.0.3.0/24", "10.0.4.0/24"]
        }
        b = {
          enable_natgw = true
          private = ["10.0.5.0/24", "10.0.6.0/24", "10.0.7.0/24"]
          public  = ["10.0.8.0/24", "10.0.9.0/24"]
        }
        c = {
          private = ["10.0.10.0/24", "10.0.11.0/24", "10.0.12.0/24"]
          public  = ["10.0.13.0/24", "10.0.14.0/24"]
        }
      }
      name    = "app"
      network = "10.0.0.0/20"
    },
    {
      azs = {
        a = {
          private = ["172.16.0.0/24", "172.16.1.0/24", "172.16.2.0/24"]
          public  = ["172.16.3.0/28", "172.16.4.0/28"]
        }
        b = {
          enable_natgw = true
          private = ["172.16.5.0/24", "172.16.6.0/24", "172.16.7.0/24"]
          public  = ["172.16.8.0/28", "172.16.9.0/28"]
        }
        c = {
          private = ["172.16.10.0/24", "172.16.11.0/24", "172.16.12.0/24"]
          public  = ["172.16.13.0/28", "172.16.14.0/28"]
        }
      }
      name    = "cicd"
      network = "172.16.0.0/20"
    },
    {
      azs = {
        a = {
          enable_natgw = true
          private = ["192.168.0.0/24", "192.168.1.0/24", "192.168.2.0/24"]
          public  = ["192.168.13.0/28"]
        }
        b = {
          private = ["192.168.5.0/24", "192.168.6.0/24", "192.168.7.0/24"]
          public  = ["192.168.8.0/28"]
        }
        c = {
          private = ["192.168.10.0/24", "192.168.11.0/24", "192.168.12.0/24"]
          public  = ["192.168.13.0/28"]
        }
      }
      name    = "general"
      network = "192.168.0.0/20"
    }
  ]
}

module "vpcs" {
  source = "git@github.com:JudeQuintana/terraform-modules.git//networking/tiered_vpc_ng?ref=v1.4.5"

  for_each = { for t in local.vpc_tiers : t.name => t }

  env_prefix       = var.env_prefix
  region_az_labels = var.region_az_labels
  tier             = each.value
}

Intra VPC Security Group Rules

The Intra VPC Security Group Rule module will create an inbound only rule for each VPC that will point to all other VPC networks except the VPC itself.

This will allow ssh and ping protocols accross Tiered VPCs so communication will just work once we enable routing to each other with Transit Gateway.

# snipppet
locals {
  intra_vpcs_security_group_rules = [
    {
      label     = "ssh"
      from_port = 22
      to_port   = 22
      protocol  = "tcp"
    },
    {
      label     = "ping"
      from_port = 8
      to_port   = 0
      protocol  = "icmp"
    }
  ]
}

module "intra_vpc_security_group_rules" {
  source = "git@github.com:JudeQuintana/terraform-modules.git//networking/intra_vpc_security_group_rule_for_tiered_vpc_ng?ref=v1.4.5"

  for_each = { for r in local.intra_vpcs_security_group_rules : r.label => r }

  env_prefix = var.env_prefix
  vpcs       = module.vpcs
  rule       = each.value
}

It’s important that the Security Group rules are created in a separate module from the routing.

Transit Gateway Centralized Router

The Transit Gateway Centralized Router module will attach all AZs in each Tiered VPC to a TGW.

Think router on a stick!

All VPC attachments will be associated and routes propagated to one TGW Route Table.

Each Tiered VPC will have all their route tables updated with a generated route to all other VPC networks via the TGW.

# snippet
module "tgw_centralized_router" {
  source = "git@github.com:JudeQuintana/terraform-modules.git//networking/transit_gateway_centralized_router_for_tiered_vpc_ng?ref=v1.4.5"

  env_prefix       = var.env_prefix
  region_az_labels = var.region_az_labels
  amazon_side_asn  = 64512
  vpcs             = module.vpcs
}

I didn’t add any TGW peering, cross-account/cross-region sharing, or specific TGW routes (ie aws_ec2_transit_gateway_route) at this time.

I’m still learning how to put TGW sharing together for the next iteration.

Caveats

The modules build resources that will cost some money but should be minimal for the demo. (ie NATGW, EIP, TGW)

You wont be able to see the generated pet names for subnet naming on plan, only on apply.

  • Even though you can delete subnets in a VPC, remember that the NAT Gateways get created in the first public subnet in the list for the AZ.
  • The Centralized Router uses the public first public subnet in each AZ for each the VPC attachment because a public subnet will always exist in a Tiered VPC.

When modifying an AZ or VPCs in an existing configuration with A TGW Centralized rouer:

  • Adding
    • The VPCs must be applied first.
    • Then apply Intra Security Groups Rules and TGW Centralized Router.
  • Removing
    • The first public subnet in the AZ being removed must be manually removed (modified) from the TGW VPC attachments first before applying to the VPC, SG Rules, and TGW.
    • Otherwise, Terraform wants to remove the attachment from the TGW after the subnet is deleted but the subnet can’t be deleted until it’s attachment is removed from the TGW (strange circular dependency that I haven’t figured out).
    • Full teardown (destroy) works fine.

Update January 29th 2023:

The Shokunin version v1.4.6 has been released for the modules above.

New features means new steez in Slappin chrome on the WIP!

The demo below has been updated to use Shokunin version v1.4.6 (and later).

Trifecta Demo Time

This will be a demo of the following:

  • Configure us-west-2a and us-west-2b AZs in app VPC - 10.0.0.0/20
    • Launch app-public instance in public subnet.
  • Configure us-west-2b AZ with NATGW in cicd VPC - 172.16.0.0/20
    • Launch cicd-private instance in private subnet.
  • Configure us-west-2c AZ in general VPC - 192.168.0.0/20
    • Launch general-private instance in private subnet.
  • Configure security groups for access across VPCs.
    • Allow ssh and ping.
  • Configure routing between all public and private subnets accross VPCs via TGW.
  • Verify connectivity with t2.micro EC2 instances.
  • Minimal assembly required.

Pre-requisites:

  • git
  • curl
  • Terraform 1.3.0+
  • Pre-configured AWS credentials
    • An AWS EC2 Key Pair should already exist in the us-west-2 region and the private key should have user read only permissions.
      • private key saved as ~/.ssh/my-ec2-key.pem on local machine.
      • must be user read only permssions chmod 400 ~/.ssh/my-ec2-key.pem

Assemble the Trifecta by cloning the Networking Trifecta Demo repo.

$ git clone git@github.com:JudeQuintana/terraform-main.git
$ cd networking_trifecta_demo

Update the key_name in variables.tf with the EC2 key pair name you’re using for the us-west-2 region (see pre-requisites above).

# snippet
variable "base_ec2_instance_attributes" {
  ...
  default = {
    key_name      = "my-ec2-key"            # EC2 key pair name to use when launching an instance
    ami           = "ami-01badf1deffd96f68" # AWS Linux 2 us-west-2 x86
    instance_type = "t2.micro"
  }
}

The VPCs must be applied first:

$ terraform init
$ terraform apply -target module.vpcs

Now we’ll:

  • Build security groups rules to allow ssh and ping across VPCs.
  • Launch instances in each enabled AZ for all VPCs.
  • Route between VPCs via TGW.
$ terraform apply -target module.intra_vpc_security_group_rules -target aws_instance.instances -target module.centralized_router

Once the apply is complete, it will take 1-2 minutes for the TGW routing to fully propagate.

Verify Connectivity Between VPCs

$ chmod u+x ./scripts/get_instance_info.sh
$ ./scripts/get_instance_info.sh

Example output:

# aws_instance.instances["cicd-private"]
    private_ip                           = "172.16.5.11"

# aws_instance.instances["general-private"]
    private_ip                           = "192.168.10.8"

# aws_instance.instances["app-public"]
    private_ip                           = "10.0.3.200"
    public_ip                            = "54.187.241.115"

# module.vpcs["app"].aws_vpc.this
    default_security_group_id        = "sg-id-1234"

# My Public IP
XX.XXX.XXX.XX

# If you have awscli configured follow the instructions below otherwise you have to do it manually in the AWS console
# AWS CLI Command to copy, replace both app-vpc-default-sg-id and My.Public.IP.Here and run script:

aws ec2 authorize-security-group-ingress --region us-west-2 --group-id app-vpc-default-sg-id --protocol tcp --port 22 --cidr My.Public.IP.Here/32

Run the awscli command from the output above to add an inbound ssh rule from “My Public IP” to the default security group id of the App VPC.

Next, ssh to the app-public instance public IP (ie 54.187.241.115) using the EC2 key pair private key.

Then, ssh to the private_ip of the general-private instance, then ssh to cicd-private, then ssh back to app-public.

$ ssh -i ~/.ssh/my-ec2-key.pem -A ec2-user@54.187.241.115

[ec2-user@app-public ~]$ ping google.com # works! via igw
[ec2-user@app-public ~]$ ping 192.168.10.8 # works! via tgw
[ec2-user@app-public ~]$ ssh 192.168.10.8

[ec2-user@general-private ~]$ ping google.com # doesn't work! no natgw
[ec2-user@general-private ~]$ ping 172.16.5.11 # works! via tgw
[ec2-user@general-private ~]$ ssh 172.16.5.11

[ec2-user@cicd-private ~]$ ping google.com # works! via natgw
[ec2-user@cicd-private ~]$ ping 10.0.3.200 # works! via tgw
[ec2-user@cicd-private ~]$ ssh 10.0.3.200

[ec2-user@app-public ~]$

🔻 Trifecta Complete!!!

Clean Up

$ terraform destroy

Final Thoughts

This article endeding up being way more work than I thought was needed so thanks for taking the time to read about my random cloud networking ideas, stay up!

~jq1

Feedback

What did you think about this post? jude@jq1.io

Synthesizing Tiered VPC in Terraform Terraform Opinion #23: Use list of objects over map of maps