Module versioning and CI/CD considerations

[sfenman]

Hey everyone! I started experimenting with Terramate some days ago and so far I really like the experience.
My structure has base modules in different repositories and then a mono-repo similar to your reference repo.

In the modules directories I have several different opinionated modules referencing my base modules alongside with some data_sources to deploy into different projects and stacks.

While working with that, two questions came to my mind:

1. Not too related to Terramate itself but more related to the module structure and versioning. What is a good way to use different
   versions of my modules inside the stacks? I created a global variable that can be used as the module version so now every production stack can override it while testing,staging stacks can have the default global.module_version. This way I can apply all stacks non-prod stacks at once while requiring to "cherry pick" the version changes for production stacks. I would love to see other implementations about that.
2. What's your opinion about CI/CD systems and integration with Terramate? Of course one could use the same configuration of
   their CD system as they would do with plain Terraform leveraging only the code generation and global variables features (which
   are already enough for someone to adopt Terramate) but also terramate run command along with stack execution order are also
   too good to ignore. I would really like your thoughts on this as it is one of the biggest pain points I have right now with other
   terraform related tools

[soerenmartius]

Hi @sfenman,

Thanks for reaching out! While I will leave your first question for @mariux to answer, let me comment on your second question.

What's your opinion about CI/CD systems and integration with Terramate? Of course one could use the same configuration of
their CD system as they would do with plain Terraform leveraging only the code generation and global variables features (which
are already enough for someone to adopt Terramate) but also terramate run command along with stack execution order are also
too good to ignore. I would really like your thoughts on this as it is one of the biggest pain points I have right now with other
terraform related tools

We are working on a GitHub Action that enables you to easily implement Terramate into your existing CI/CD workflows in GHA. In addition, we are working on some guides also that will elaborate on how to use Terramate correctly when working in CI/CD. We plan to ship the first guide in May together with the release of the upcoming Terramate website.

I'd love to understand which CI/CD solution you are using so we can consider it for the guides right away. Feel free to join our discord community also.

[sfenman]

Thanks for the reply @soerenmartius ,

Right now I'm evaluating various managed platforms alternatives to TFC. The thing is that almost all of these platforms have a "stack" concept similar to the stacks in Terramate. They also have some detect changes mechanisms to run plan and apply commands that are contradicting with the change detection of Terramate and the terramate run commands.

Ultimately, it may come down to personal preference and opinion when deciding which method to choose. However, I would appreciate any insights, considerations, and gotchas regarding these two methods to aid me to make a decision.

[soerenmartius]

Thanks for your feedback! I will discuss this with the team and get back to you. In the meantime, I am also happy to share that we are working on Terramate Cloud. While CI/CD will not be shipped with the initial release, we expect the beta to launch in June.

[soerenmartius]

@sfenman could you share with me the platforms you are evaluating? I will look into those to evaluate how Terramate can help. However, we implemented large platforms using GitHub Actions by fully leveraging Terramate features such as change detection. Preview comments in PRs are very helpful also. I will get an example prepared to share with you next week.

[sfenman]

Currently evaluating Spacelift. Using Github Actions without the need of an external platform would be an ideal scenario, however I have to manually implement things like race conditions in multiple PRs, team access management, preview comments and Atlantis-style workflows that work well with large plan files etc etc.

[mariux]

Hi @sfenman,

Thank you for your feedback.

1. Not too related to Terramate itself but more related to the module structure and versioning. What is a good way to use different
   versions of my modules inside the stacks? I created a global variable that can be used as the module version so now every production stack can override it while testing, and staging stacks can have the default `global.module_version`. This way I can apply all stacks non-prod stacks at once while requiring to "cherry-pick" the version changes for production stacks. I would love to see other implementations of that.

I am not sure I fully understand the need to "cherry-pick" the versions for production. Maybe you could share some more details about the problem you have.

In general, you can define Terramate globals on multiple levels in the hierarchy. The higher up the hierarchy the more stacks will inherit such a setting.

In addition, you can make such a decision conditional by making it dependent on another variable and just set the value in a central place on a very high level in the hierarchy:

Based on an environment:

globals {
  module_version = global.env == "production" ? "1.0 : "2.0-alpha"
}

or based on a specific stack path:

globals {
  module_version = terramate.stack.path.relative == "/path/to/stack" ? "2.0-alpha" : "1.0"
}

The same is possible for all combinations of regular expressions with tm_regex(), using various stack metadata like terramate.stack.id or terramate.stack.tags.

As globals are lazily evaluated on the stack none of the variables need to exist on the higher level, but each stack reachable from the config needs to be able to evaluate all globals used or safeguard with tm_try() and tm_can().

[sfenman]

You answered my question perfectly, thank you.