Welcome to the next episode of thinking out loud to route in the cloud, you know my steez!
Earlier this season we abstracted a TGW Centralized Router.
Part of it’s responsibility is to manage routes between Tiered VPCs within a single region.
It will create routes to other VPC networks in private and public route tables for each VPC (except itself).
# snippet
variable "vpcs" {
type = map(object({
network = string
az_to_private_route_table_id = map(string)
az_to_public_route_table_id = map(string)
}))
}
locals {
# { vpc-1-network => [ "vpc-1-private-rtb-id-1", "vpc-1-public-rtb-id-1", ... ], ...}
vpc_network_to_private_and_public_route_table_ids = {
for vpc_name, this in var.vpcs :
this.network => concat(values(this.az_to_private_route_table_id), values(this.az_to_public_route_table_id))
}
# [ { rtb_id = "vpc-1-rtb-id-123", other_networks = [ "other-vpc-2-network", "other-vpc3-network", ... ] }, ...]
associate_private_and_public_route_table_ids_with_other_networks = flatten(
[for network, rtb_ids in local.vpc_network_to_private_and_public_route_table_ids :
[for rtb_id in rtb_ids : {
rtb_id = rtb_id
other_networks = [for n in keys(local.vpc_network_to_private_and_public_route_table_ids) : n if n != network]
}]])
# { rtb-id|route => route, ... }
private_and_public_routes_to_other_networks = merge(
[for r in local.associate_private_and_public_route_table_ids_with_other_networks :
{ for rtb_id_and_route in setproduct([r.rtb_id], r.other_networks) :
format("%s|%s", rtb_id_and_route[0], rtb_id_and_route[1]) => rtb_id_and_route[1] # each key must be unique, dont group by key
}]...)
}
resource "aws_route" "this" {
for_each = local.private_and_public_routes_to_other_networks
destination_cidr_block = each.value
route_table_id = split("|", each.key)[0]
transit_gateway_id = aws_ec2_transit_gateway.this.id
# make sure the tgw route table is available first before the setting routes on the vpcs
depends_on = [aws_ec2_transit_gateway_route_table.this]
}
It’s possible to calculate only private or only public routes to other VPCs by passing in an empty list for subnets that you don’t want processed.
Looking forward, it would be nice to route between many TGW Centralized Routers but cross region with some kind of TGW Super Router abstraction.
I think it should roughly look like:
# enable cross region routing between the TGW Centralized Routers
module "tgw_super_router" {
source = "git/path/to/tgw_super_router_for_tgw_centralized_router"
providers = {
aws.local = aws.use1 # build super router in aws.local provider
aws.remote = aws.usw2
}
env_prefix = var.env_prefix
region_az_labels = var.region_az_labels
local_tgw_centralized_routers = [module.tgw_centralized_router_use1] # same region/provider as super router
remote_tgw_centralized_routers = [module.tgw_centralized_router_usw2] # separate var for dealing with different region/provider
tgw_routes = [tgw_route_objects_go_here] # additional blackhole and override routes?
}
TGW Super Router behavior assumptions:
- No cross account sharing just yet.
- Manage cross region routes between many Centralized Routers.
- Ability to inject blackhole and override routes.
- Includes TGW Peering between Centralized Routers (???)
- Manage cross region routes between many Tiered VPCs.
- Similar to how Centralized Routers currently add routes to each Tiered VPC.
- …
We can already start to see that the process of adding routes to each VPC can be used in more than one place.
It seems like good time to encapsulate the route generation by moving it into it’s own Generate Routes to Other VPCs module to keep it DRY for reuse.
This will give us chance to utilize the experimental terraform test framework to build a test suite for our new function.
For instructions on running this generate routes Terraform test suite go here.
We can add more gating with CIDR validation at the variable.tf level and ensuring only a map of tierd_vpc_ng objects is passed.
We’ll use call for the output to treat this module object as a function (with no resources) like an execute function call.
resource "test_assertions" "generate_routes_to_other_vpcs" {
component = "generate_routes_to_other_vpcs"
equal "map_of_unique_routes_to_other_vpcs" {
description = "generated routes"
got = module.main.call
want = local.private_and_public_routes_to_other_vpcs
}
}
The output is structured like { "rtb-id|route" => "route", ... }.
Route resource consumers need to split on the key with the pipe character “|” to get the route table id.
It’s a shortcut with a consequence that consumers need to how to handle the data in the map (ie split on the key) because the route resource only needs two types of data for my use case.
Now we can replace adding routes to each VPC in the Centralized Router in the future with:
# snippet
variable "vpcs" {
type = map(object({
network = string
az_to_private_route_table_id = map(string)
az_to_public_route_table_id = map(string)
}))
}
module "generate_routes_to_other_vpcs" {
source = "git@github.com:JudeQuintana/terraform-modules.git//utils/generate_routes_to_other_vpcs?ref=v1.3.0"
vpcs = var.vpcs
}
resource "aws_route" "this" {
for_each = module.generate_routes_to_other_vpcs.call
destination_cidr_block = each.value
route_table_id = split("|", each.key)[0]
transit_gateway_id = aws_ec2_transit_gateway.this.id
# make sure the tgw route table is available first before the setting routes on the vpcs
depends_on = [aws_ec2_transit_gateway_route_table.this]
}
We could build different data structures inside the module to better organize the output to potentially remove the need for the route consumer know how to operate on the map other than iterate.
It would be another layer of iteration and probably worth experimenting further.
There are several ways to present data structures needed by route consumers so having tests makes it easier to refactor.
Now that this Generate Routes to Other VPCs module is complete, I can start thinking about what should be implemented next for the TGW Super Router!
Stay inspired and keep building.
Until the next episode…#AoD
~jq1
Super helpful links:
- Module Testing Experiment by HashiCorp.
- Terraform Module Testing by Ned In The Cloud.
- Integration Testing with Terraform 0.15 by Marko Bevc.
- Special thanks to Marko for helping me get my test suite running for this article!
Update:
As it turns out, a list of route objects makes it easier to handle when passing them to other route resource types (ie vpc, tgw) instead of using a map that describes a route in terms that are too generic.
The caller should build their own map with unique keys before passing it to an aws_route.
So I refactored the call output to reflect that.
# snippet
variable "vpcs" {
type = map(object({
network = string
az_to_private_route_table_id = map(string)
az_to_public_route_table_id = map(string)
}))
}
module "generate_routes_to_other_vpcs" {
source = "git@github.com:JudeQuintana/terraform-modules.git//utils/generate_routes_to_other_vpcs?ref=v1.4.1"
vpcs = var.vpcs
}
locals {
vpc_routes_to_other_vpcs = {
for this in module.generate_routes_to_other_vpcs.call :
format("|", this.route_table_id, this.destination_cidr_block) => this
}
}
resource "aws_route" "this" {
for_each = local.vpc_routes_to_other_vpcs
destination_cidr_block = each.value.destination_cidr_block
route_table_id = each.value.route_table_id
transit_gateway_id = aws_ec2_transit_gateway.this.id
# make sure the tgw route table is available first before the setting routes on the vpcs
depends_on = [aws_ec2_transit_gateway_route_table.this]
}
You can still get the legacy map of generically defined routes with the call_legacy
output.
But now I think generating a map of generically defined routes with unique keys for the caller is not a shortcut worth taking becuase of the inflexibility when needing different transforms.
# snippet
variable "vpcs" {
type = map(object({
network = string
az_to_private_route_table_id = map(string)
az_to_public_route_table_id = map(string)
}))
}
module "generate_routes_to_other_vpcs" {
source = "git@github.com:JudeQuintana/terraform-modules.git//utils/generate_routes_to_other_vpcs?ref=v1.4.1"
vpcs = var.vpcs
}
resource "aws_route" "this" {
for_each = module.generate_routes_to_other_vpcs.call_legacy
destination_cidr_block = each.value
route_table_id = split("|", each.key)[0]
transit_gateway_id = aws_ec2_transit_gateway.this.id
# make sure the tgw route table is available first before the setting routes on the vpcs
depends_on = [aws_ec2_transit_gateway_route_table.this]
}
Also, I didn’t end up using module.generate_routes_to_other_vpcs
function in more than one place like I thought I would but was still useful in Centralized Router.
Update January 29th 2023:
The Shokunin version v1.4.6 (and later) has been released for Generate Routes to Other VPCs module.
Refactored the interface but has same behavior.
Before:
variable "vpcs" {
description = "map of tiered_vpc_ng objects"
type = map(object({
network = string
az_to_private_route_table_id = map(string)
az_to_public_route_table_id = map(string)
}))
}
After:
variable "vpcs" {
description = "map of tiered_vpc_ng objects"
type = map(object({
network_cidr = string
private_route_table_ids = list(string)
public_route_table_ids = list(string)
}))
}
New features means new steez in Slappin chrome on the WIP!