Archetype Terraform module for the Private Module Registry

This repository is intended to help you get started writing Terraform modules for use with SG1 and the Terraform Private Module Registry quickly and easily.

Usage

To make use of this, you should:

  • Create a new repository in the CXEPI GitHub organisation and choose to use this repository as the template
  • Edit the settings for the newly-created repository and update the Collaborators and teams
    • Add the user swtg-robot.gen with Write permissions (for automated versioning commits).
    • Add the user cx-terraform-cloud with Admin permissions (for enabling them to be added to the Terraform Cloud Private registry).
    • Add the Team SG1 Admins with Admin permissions (for the SG1 team to make admin changes if required).
    • Add the Team sg1-write with Write permissions (for the SG1 collaborators to create PRs).
    • Plus add your respective team(s) to the repository with appropriate permissions.
  • Add appropriate branch protections
  • git clone your new repository to your local system
  • Write your Terraform code in the root directory of this module however you see fit
    • Remove the EXAMPLE.tf file and the example-submodule directory when you are comfortable, these are simply example code to demonstrate how things hook together in Terraform
    • when writing the new module, do not include a provider block or any backend information; the provider should come from the calling workspace and the backend is implicitly configured on the workspace by Terraform Cloud
  • (optional, but highly recommended) write Terratest tests
    • tests are contained in the terratest subdirectory
    • the included example_test.go demonstrates the overall approach
      • Rename example_test.go to something more descriptive, retaining the _test suffix
      • Rename the function from TestExample to something more appropriate, retaining the Test prefix
      • More functions can be added but be aware that Terratest uses a local state file, so to run parallel tests, each test needs to copy the Terraform code into a unique, temporary location
      • Edit the terraformOptions object to include a correct set of input variables for your module/test. For running parallel tests, this should be declared within each function so that the variable scope is correct
      • To run tests locally you will need Terraform, AWSCLI and Go
      • brew install terraform
      • brew install awscli
      • brew install golang
  • Install the pre-commit hooks onto your local system:
    • cd to the root of the repository
    • brew install pre-commit
    • pre-commit install --install-hooks
  • Make sure there is a release.version file in the root of the repository containing a major.minor.patch version (for example 1.0.0).
    • During the build, the version in release.version is used to create a new tag in the repository
    • After this, the version in release.version is incremented by one patch version and then committed to the main/master branch in preparation for the next main/master build
    • Anything more than patch version increments are manual; see Versioning of the module
  • Push your changes back to a branch in the repository and open a Pull Request

Enabling CI / running of tests

  • On the CircleCI Add Projects screen for our organisation you should see your new repository listed, with a “Set Up Project” button
  • Push the button, which should install webhooks and deploy keys into your new repository’s settings
  • You should shortly see a test run appear on the CircleCI dashboard
  • You will be required to add the swtg-robot.gen private key (with the fingerprint 12:c7:63:86:0e:d8:75:9a:17:ad:a9:16:5b:5d:94:41, this is not the same as the previous GitHub Enterprise on-prem key) to the CircleCI job (under Project Settings, SSH Keys)
    • request access to the CYB-SAFE.CXEPI-DevTools cisco group
    • on cpim.cisco.com , find the swtg-robot.gen On git key
    • on your CCI project config, SSH keys, add new key for github.com, paste in the above key
  • On all branches, this will perform some Terraform linting, sanity checks, and run any provided Terratest tests
  • If this build is running from the main branch then once the steps have passed
    • the release.version value to be incremented by 0.0.1
    • the repository is tagged with this new version
    • the file is committed back to the repository with the [skip-ci] comment which stops another build from happening

Publishing Your Module to the Private Module Registry in Terraform Cloud

Note

  • There is a circle CI job associated with this module – https://app.circleci.com/pipelines/github/CXEPI/terraform-tfe-archetype?branch=main&filter=all
  • To test any changes/additions to the repository it is advised to create a new repository from this template, then merge the changes back to this repository, please ensure that the tests are still passing before you merge.
  • You need to change the context for the job cx-cloud-build/terratest-job in ./.circleci/config.yaml. This context needs to match the creds required by your tests, these currently are;
    • sre-sandbox-terratest-context This context has credentials for interacting with the aws-account-sre user in the cx-nprd-sre-sandbox AWS account.
    • platform-deploy This context has the TFE_TOKEN which allows tests to be run against TFC.

Best Practices

Versioning of the module

TBD Some caveats probably exist here that I haven’t thought of…

The Hashicorp Private Module Registry requires the Semantic Versioning format to be used, which makes a sensible follow-on that we should try to actually use Semantic Versioning semantics (rather than just changing an arbitrary two-digit number to a three-digit number)

The “API” of a module can largely be defined by its inputs and outputs and these will provide guidance on how big a version bump to do. For example:

  • If the inputs/outputs do not change but you have changed some internal functionality of the module, to either refactor the code, or make a change that should become a default then this probably constitutes a PATCH version increment.
  • If the inputs change to add some new variables with default values (i.e. that don’t need to be specified when instantiating it) then this probably constitutes a MINOR version increment
  • Or if you remove a variable and make a previously-variable value a forced default, this probably constitutes a MINOR version increment (because the calling workspace/module can still provide the variable value, but it will simply be ignored)
  • Similarly if the outputs change to add a new output, then because an output can be left “unconsumed” by the calling workspaces/modules, this probably constitutes a MINOR version increment
  • If you add a new input that cannot be defaulted, or you perform major changes to the resources created by a module, this should be a MAJOR version increment
  • If the minimum provider version needs bumping because support is needed for a new resource or attribute this should be a MAJOR version increment

Provider version constraints

Within modules, do not be too strict on version constraints. Only enforce the minimum version which is required for the resources being created, and leave any stricter constraints to the calling workspace. This ties in with Hashicorp’s recommendations.

For example, for a workspace containing two modules, with the following version constraints:

  • workspace: ~> 4.0
  • module1: ~> 3.0.0
  • module2: ~> 4.0.0

This workspace cannot run because the constraints are not satisfiable (the workspace requires “a non-beta provider of version 4.x“, which is incompatible with module1 because it requires “a non-beta provider of version 3.0.x“)

A better approach is to enforce only minimum versions in the modules, making the constraints:

  • workspace: ~> 4.0
  • module1: >= 3.0.0
  • module2: >= 4.0.0

In this second case, the workspace requirement is the same (“a non-beta provider of version 4.x“) however this can now be satisfied on both modules because they only enforce minimum versions, and the workspace constraint will allow use of 4.0, 4.1, 4.2 and so on over time, but not 5.0.

Provider configuration

This shouldn’t be declared in the module, but passed in from the workspace to allow more flexibility (i.e. not tying any module to any specific provider config). Again, this is a Hashicorp recommendation.

Branch protections

Ensure that you cannot merge to the main branch without a PR with the following settings:

  • Branch name pattern: main
  • Require a pull request before merging:
    • Require approvals: 1
    • Dismiss stale pull request approvals when new commits are pushed: true
    • Require review from Code Owners: true
    • Allow specified actors to bypass pull request requirements: swtg-robot-gen
  • Include administrators: true
  • Restrict who can push to matching branches: swtg-robot-gen

Also in the main repository settings there is an option to automatically delete incoming branches after pull requests are merged. This is on the main repository settings page (not under branch protections):

  • Automatically delete head branches: true

Pre-commit hooks

Install the pre-commit hooks, this gives you quick feedback about the quality of your code. More information here, and the .pre-commit-config.yaml in the repository root contains information about the specific checks that are enabled

Terratests

Terratests are a way to reliably ensure that your module manages resources based on argument inputs. It is primarily meant to test out logic within the module which may or may not change the output of the module. We can leverage Terratest to repeatedly get immediate feedback loops on complex infrastructure that otherwise would be missed. As the module matures, additional tests can be added for further resilience when upgrading dependencies or changing your IaC.

Prior to writing out the Golang code for testing, it’s important to write your test Terraform code in the examples/ folder, ensuring that each folder underneath it has a meaningful title. By having an examples/ folder, it will be read in the Terraform Cloud Private Module Registry as a location for consumers of the module to look at for actual usage. This will also ensure that as new features or fixes are introduced to the module, the examples are kept up to date as it’s required to run as part of the CI process. Note that within each example, you should be able to run this either via Terratest or manually. This also means that the example should run independently from other examples.

  • Start by the simplest example: Test if your module will create what it is intended to create. You can point to the examples/ folder using
exampleFolder := test_structure.CopyTerraformFolderToTemp(t, "../", "examples/workspace_with_vcs")
  • Always ensure to run cleanup so we don’t have wasted infrastructure sitting around. We defer executing the destroy function until the end of the tests. Refer to the archetype terratest example. Below is the basic guideline:
    • terraformOptions := terraform.WithDefaultRetryableErrors(t, &terraform.Options)
    • defer terraform.Destroy(t, terraformOptions)
    • terraform.InitAndApplyAndIdempotent(t, terraformOptions)
    • The exact implementation may differ for a specific testcase, but the correct order of operations is:
      • Define your terraform.Options object (which includes the path with the code, any variables, etc)
      • As soon as the object is defined, defer the destruction of it, which will ensure that the cleanup runs before the thread exits
      • Once the destroy operation is on the defer stack, immediately before the thread exits (whether the function was overall successful or not) a terraform destroy will be run to tear down anything that was deployed
  • Depending on the complexity you may need to increase the default duration of your local tests to 30m. The default is set at 10m and will forcibly kill your tests or even skip clean up if that expires. Be careful on increasing the time in circleci jobs as this will affect your build time.
    • go test -timeout 30m
    • If running on your circleci job you’ll want to add a reference to no_output_timeout in this run step to override the default job step timeout of 10 mins and also consider this in terms of total job run timeout limits.
  • Consider turning off Go caching for test results to ensure tests are run on every go test command. You can do this by setting the count flag to 1
    • go test example_test.go -count=1 -timeout 30m
  • Ensure your terraform configuration is idempotent with a full cycle being init/apply/plan resulting in zero changes on the plan step (meaning there are no difference between your desired state and actual state). You can do this by using:
    • terraform.InitAndApplyAndIdempotent(t, terraformOptions)

Checkout GruntWork’s best practices with using terratest here


Requirements

Name Version
null ~> 3.0

Providers

Name Version
null 3.1.0

Modules

Name Source Version
example_submodule ./modules/submodule n/a

Resources

Name Type
null_resource.null_resource resource
null_data_source.null_data_source data source

Inputs

Name Description Type Default Required
input_number starting number; we’ll add 5 to this number n/a yes
input_string an example string string n/a yes

Outputs

Name Description
from_data_source mirrors the input stringVerified by Terratest: no
number_plus_5 outputs input_number plus 5Verified by Terratest: yes
uppercased_string outputs the input string, through upper()Verified by Terratest: yes

GitHub

View Github