Go Modules Replace Use Case: Terratest and LocalStack

At work we’re planning on having our IaC to grow up a bit. We’re really wanting it to get a job and start contributing more around here. Something like a shiny new Terraform Pipeline would be nice. That means we need to get our Terraform testing sorted out.

I came across this HashiCorp video called Testing Infrastructure as Code on Localhost where Samuel Kihahu talks about combining Terratest + Terraform + LocalStack to allow him to test Terraform modules locally. Pretty F’n cool, right?

I was able to get the latest (v0.11.5) LocalStack Docker container up and running relatively quickly.

$ pip install localstack
$ localstack start

Then I configured a Terraform provider with custom service endpoints in an example s3 bucket module (with some modifications) to point to the LocalStack endpoints running on localhost.

terraform {
  required_version = ">= 0.12.29"
  required_providers {
    aws = "2.70.0"
  }
}

provider "aws" {
  region                      = "us-east-1"
  s3_force_path_style         = true
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true

  endpoints {
    apigateway     = "http://localhost:4566"
    cloudformation = "http://localhost:4566"
    cloudwatch     = "http://localhost:4566"
    dynamodb       = "http://localhost:4566"
    es             = "http://localhost:4566"
    firehose       = "http://localhost:4566"
    iam            = "http://localhost:4566"
    kinesis        = "http://localhost:4566"
    lambda         = "http://localhost:4566"
    route53        = "http://localhost:4566"
    redshift       = "http://localhost:4566"
    s3             = "http://localhost:4566"
    secretsmanager = "http://localhost:4566"
    ses            = "http://localhost:4566"
    sns            = "http://localhost:4566"
    sqs            = "http://localhost:4566"
    ssm            = "http://localhost:4566"
    stepfunctions  = "http://localhost:4566"
    sts            = "http://localhost:4566"
    ec2            = "http://localhost:4566"
  }
}
...

Note: I’m using Terraform 0.12 with the latest 2.x provider. I’ve tried Terraform 0.13 with the latest 3.x provider and LocalStack started throwing errors.

I was able to manually run terraform apply on the s3 bucket module and it successfully created a mocked s3 bucket resource in LocalStack! EZ PZ!

Next I installed Golang 1.15.2 then I pulled up the Terratest Quick start guide and followed the 5 short steps under Setting up your project.

Using the associated example s3 bucket test and making a few modifications:

package test

import (
	"fmt"
	"strings"
	"testing"

	"github.com/gruntwork-io/terratest/modules/aws"
	"github.com/gruntwork-io/terratest/modules/random"
	"github.com/gruntwork-io/terratest/modules/terraform"
	"github.com/stretchr/testify/assert"
)

// An example of how to test the Terraform module in examples/terraform-aws-s3-example using Terratest.
func TestTerraformAwsS3Example(t *testing.T) {
	t.Parallel()


	// Give this S3 Bucket a unique ID for a name tag so we can distinguish it from any other Buckets provisioned
	// in your AWS account
	expectedName := fmt.Sprintf("terratest-aws-s3-example-%s", strings.ToLower(random.UniqueId()))

	// Give this S3 Bucket an environment to operate as a part of for the purposes of resource tagging
	expectedEnvironment := "Automated Testing"

	// This usually picks a random AWS region to test in. This helps ensure your code works in all regions.
	// I used us-east-1 only since the default region for LocalStack is us-east-1
	awsRegion := aws.GetRandomStableRegion(t, []string{"us-east-1"}, nil)

	terraformOptions := &terraform.Options{
		// The path to where our Terraform code is located
		TerraformDir: "../examples/s3_bucket",

		// Variables to pass to our Terraform code using -var options
		Vars: map[string]interface{}{
			"tag_bucket_name":        expectedName,
			"tag_bucket_environment": expectedEnvironment,
			"with_policy":            "true",
		},

		// Environment variables to set when running Terraform
		EnvVars: map[string]string{
			"AWS_DEFAULT_REGION": awsRegion,
		},
	}

	// At the end of the test, run `terraform destroy` to clean up any resources that were created
	defer terraform.Destroy(t, terraformOptions)

	// This will run `terraform init` and `terraform apply` and fail the test if there are any errors
	terraform.InitAndApply(t, terraformOptions)

	// Run `terraform output` to get the value of an output variable
	bucketID := terraform.Output(t, terraformOptions, "bucket_id")

	// Verify that our Bucket has versioning enabled
	actualStatus := aws.GetS3BucketVersioning(t, awsRegion, bucketID)
	expectedStatus := "Enabled"
	assert.Equal(t, expectedStatus, actualStatus)

	// Verify that our Bucket has a policy attached
	aws.AssertS3BucketPolicyExists(t, awsRegion, bucketID)
}

When I run go test, go will install the dependencies for Terratest and will run the terraform apply command, verify the s3 bucket resource was created then terraform destroy.

As expected the s3 bucket resources will be successfully created and destroyed in LocalStack but the test will fail because Terratest tries to verify the s3 bucket is created in AWS and not in LocalStack.

The aws package used in Terratest isn’t configurable to use custom endpoints because Terratest is normally used to verify actual AWS resources for tests. There would need to be Terratest code changes to support a custom endpoints configuration to point to LocalStack.

Fortunately, the great devs on planet earth already have a PR in place to make this happen. It looks like this solution is in a working state but how do I confirm that?

The problem I face now is, how do I point the Terratest package to use a branch, on a fork of Terratest at an earlier version, in a 3rd party repo?

After brainstorming with a co-worker, he said my problem sounded like a good use case for the go module replace directive but he doesn’t know exactly how that would work since he’s never had to do it before. I wasn’t sure either since I don’t have any go module experience.

I looked at the forked repo Terratest tags and the latest tag was v0.28.5. From the go docs the go module replace directive will take a version (or a relative path) but not a branch.

Taking a look at my go.mod file:

module github.com/GITHUB_USERNAME/REPO_NAME

go 1.15

require (
	github.com/gruntwork-io/terratest v0.30.0
	github.com/stretchr/testify v1.6.1
)

I add the replace directive with the 3rd party repos version.

module github.com/GITHUB_USERNAME/REPO_NAME

go 1.15

require (
	github.com/gruntwork-io/terratest v0.30.0
	github.com/stretchr/testify v1.6.1
)

replace github.com/gruntwork-io/terratest v0.30.0 => github.com/ffernandezcast/terratest v0.28.5

I tried running go test again but the same behavior is still occuring. Adding the v0.28.5 Terratest version for the 3rd party repo isn’t enough because the code I want is in a branch called add-aws-custom-configuration-support.

According to the answers in this stack overflow post I can use go get to get a commit hash or a branch. When I run it on the branch I want, I get an error:

$ go get github.com/ffernandezcast/terratest@add-aws-custom-configuration-support
go: github.com/ffernandezcast/terratest add-aws-custom-configuration-support => v0.28.6-0.20200915124510-25813206bebc
go get: github.com/ffernandezcast/terratest@v0.28.6-0.20200915124510-25813206bebc: parsing go.mod:
        module declares its path as: github.com/gruntwork-io/terratest
                but was required as: github.com/ffernandezcast/terratest

I’m not sure what the last two lines mean but I can see that the error gave me a psuedo version for the branch and it looks like the same version format that the replace directive should take, so I plugged it in.

module github.com/GITHUB_USERNAME/REPO_NAME

go 1.15

require (
	github.com/gruntwork-io/terratest v0.30.0
	github.com/stretchr/testify v1.6.1
)

replace github.com/gruntwork-io/terratest v0.30.0 => github.com/ffernandezcast/terratest v0.28.6-0.20200915124510-25813206bebc

Then updated my go test with the recommended variables from the PR so that we can configure the LocalStack endpoints in the aws package.

package test

import (
	"fmt"
	"strings"
	"testing"

	"github.com/gruntwork-io/terratest/modules/aws"
	"github.com/gruntwork-io/terratest/modules/random"
	"github.com/gruntwork-io/terratest/modules/terraform"
	"github.com/stretchr/testify/assert"
)

// An example of how to test the Terraform module in examples/terraform-aws-s3-example using Terratest.
func TestTerraformAwsS3Example(t *testing.T) {
	t.Parallel()

	var LocalEndpoints = map[string]string{
		"apigateway":     "http://localhost:4566",
		"cloudformation": "http://localhost:4566",
		"cloudwatch":     "http://localhost:4566",
		"dynamodb":       "http://localhost:4566",
		"es":             "http://localhost:4566",
		"firehose":       "http://localhost:4566",
		"iam":            "http://localhost:4566",
		"kinesis":        "http://localhost:4566",
		"lambda":         "http://localhost:4566",
		"route53":        "http://localhost:4566",
		"redshift":       "http://localhost:4566",
		"s3":             "http://localhost:4566",
		"secretsmanager": "http://localhost:4566",
		"ses":            "http://localhost:4566",
		"sns":            "http://localhost:4566",
		"sqs":            "http://localhost:4566",
		"ssm":            "http://localhost:4566",
		"stepfunctions":  "http://localhost:4566",
		"sts":            "http://localhost:4566",
	}
	aws.SetAwsEndpointsOverrides(LocalEndpoints)

	// Give this S3 Bucket a unique ID for a name tag so we can distinguish it from any other Buckets provisioned
	// in your AWS account
	expectedName := fmt.Sprintf("terratest-aws-s3-example-%s", strings.ToLower(random.UniqueId()))

	// Give this S3 Bucket an environment to operate as a part of for the purposes of resource tagging
	expectedEnvironment := "Automated Testing"

	// This usually picks a random AWS region to test in. This helps ensure your code works in all regions.
	// I used us-east-1 only since the default region for LocalStack is us-east-1
	awsRegion := aws.GetRandomStableRegion(t, []string{"us-east-1"}, nil)

	terraformOptions := &terraform.Options{
		// The path to where our Terraform code is located
		TerraformDir: "../examples/s3_bucket",

		// Variables to pass to our Terraform code using -var options
		Vars: map[string]interface{}{
			"tag_bucket_name":        expectedName,
			"tag_bucket_environment": expectedEnvironment,
			"with_policy":            "true",
		},

		// Environment variables to set when running Terraform
		EnvVars: map[string]string{
			"AWS_DEFAULT_REGION": awsRegion,
		},
	}

	// At the end of the test, run `terraform destroy` to clean up any resources that were created
	defer terraform.Destroy(t, terraformOptions)

	// This will run `terraform init` and `terraform apply` and fail the test if there are any errors
	terraform.InitAndApply(t, terraformOptions)

	// Run `terraform output` to get the value of an output variable
	bucketID := terraform.Output(t, terraformOptions, "bucket_id")

	// Verify that our Bucket has versioning enabled
	actualStatus := aws.GetS3BucketVersioning(t, awsRegion, bucketID)
	expectedStatus := "Enabled"
	assert.Equal(t, expectedStatus, actualStatus)

	// Verify that our Bucket has a policy attached
	aws.AssertS3BucketPolicyExists(t, awsRegion, bucketID)
}

Then I ran go test again and BAM! Terratest is now verifying the s3 bucket creation in LocalStack! Much props to the awesome devs who contributed to the PR to get this working. Hopefully, it will be merged into Terratest master branch soon!

Now, it’s time start building them local Terraform tests!

~jq1