Unable to read terraform variables.tf files into may go program - go

Question

I am attempting to write a go program that reads in a terraform variables.tf and populates a struct for later manipulation. However, I am getting errors when attempting to "parse" the file. I Am hoping someone can tell me what I am doing wrong:
Code:
package main
import (
"fmt"
"io/ioutil"
"log"
"os"
"github.com/hashicorp/hcl/v2"
"github.com/hashicorp/hcl/v2/gohcl"
"github.com/hashicorp/hcl/v2/hclsyntax"
)
type Config struct {
Upstreams []*TfVariable `hcl:"variable,block"`
}
type TfVariable struct {
Name string `hcl:",label"`
// Default string `hcl:"default,optional"`
Type string `hcl:"type"`
Description string `hcl:"description,attr"`
// validation block
Sensitive bool `hcl:"sensitive,optional"`
}
func main() {
readHCLFile("examples/string.tf")
}
// Exits program by sending error message to standard error and specified error code.
func abort(errorMessage string, exitcode int) {
fmt.Fprintln(os.Stderr, errorMessage)
os.Exit(exitcode)
}
func readHCLFile(filePath string) {
content, err := ioutil.ReadFile(filePath)
if err != nil {
log.Fatal(err)
}
fmt.Printf("File contents: %s", content) // TODO: Remove me
file, diags := hclsyntax.ParseConfig(content, filePath, hcl.Pos{Line: 1, Column: 1})
if diags.HasErrors() {
log.Fatal(fmt.Errorf("ParseConfig: %w", diags))
}
c := &Config{}
diags = gohcl.DecodeBody(file.Body, nil, c)
if diags.HasErrors() {
log.Fatal(fmt.Errorf("DecodeBody: %w", diags))
}
fmt.Println(c) // TODO: Remove me
}
ERROR
File contents: variable "image_id" {
type = string
description = "The id of the machine image (AMI) to use for the server."
sensitive = false
}
variable "other_id" {
type = string
description = "The id of the machine image (AMI) to use for the server."
sensitive = true
}
2021/03/13 19:55:49 DecodeBody: examples/string.tf:2,17-23: Variables not allowed; Variables may not be used here., and 3 other diagnostic(s)
exit status 1
Stack driver question is sadly for hcl1
Blog post I am referencing.

Answer 1

It looks like it's a bug/feature of the library, since as soon as you change string to "string", e.g.,
variable "image_id" {
type = string
...
to
variable "image_id" {
type = "string"
...
gohcl.DecodeBody succeeds.
--- UPDATE ---
So, they do use this package in Terraform, BUT they custom-parse configs, i.e., they don't use gohcl.DecodeBody. They also custom-treat type attributes by using hcl.ExprAsKeyword (compare with description). As you assumed, they do use a custom type for type, but with custom parsing you don't have to.
Below is a working example:
package main
import (
"fmt"
"log"
"os"
"github.com/hashicorp/hcl/v2"
"github.com/hashicorp/hcl/v2/gohcl"
"github.com/hashicorp/hcl/v2/hclsyntax"
)
var (
configFileSchema = &hcl.BodySchema{
Blocks: []hcl.BlockHeaderSchema{
{
Type: "variable",
LabelNames: []string{"name"},
},
},
}
variableBlockSchema = &hcl.BodySchema{
Attributes: []hcl.AttributeSchema{
{
Name: "description",
},
{
Name: "type",
},
{
Name: "sensitive",
},
},
}
)
type Config struct {
Variables []*Variable
}
type Variable struct {
Name string
Description string
Type string
Sensitive bool
}
func main() {
config := configFromFile("examples/string.tf")
for _, v := range config.Variables {
fmt.Printf("%+v\n", v)
}
}
func configFromFile(filePath string) *Config {
content, err := os.ReadFile(filePath) // go 1.16
if err != nil {
log.Fatal(err)
}
file, diags := hclsyntax.ParseConfig(content, filePath, hcl.Pos{Line: 1, Column: 1})
if diags.HasErrors() {
log.Fatal("ParseConfig", diags)
}
bodyCont, diags := file.Body.Content(configFileSchema)
if diags.HasErrors() {
log.Fatal("file content", diags)
}
res := &Config{}
for _, block := range bodyCont.Blocks {
v := &Variable{
Name: block.Labels[0],
}
blockCont, diags := block.Body.Content(variableBlockSchema)
if diags.HasErrors() {
log.Fatal("block content", diags)
}
if attr, exists := blockCont.Attributes["description"]; exists {
diags := gohcl.DecodeExpression(attr.Expr, nil, &v.Description)
if diags.HasErrors() {
log.Fatal("description attr", diags)
}
}
if attr, exists := blockCont.Attributes["sensitive"]; exists {
diags := gohcl.DecodeExpression(attr.Expr, nil, &v.Sensitive)
if diags.HasErrors() {
log.Fatal("sensitive attr", diags)
}
}
if attr, exists := blockCont.Attributes["type"]; exists {
v.Type = hcl.ExprAsKeyword(attr.Expr)
if v.Type == "" {
log.Fatal("type attr", "invalid value")
}
}
res.Variables = append(res.Variables, v)
}
return res
}
Add for completeness, example/string.tf:
variable "image_id" {
type = string
description = "The id of the machine image (AMI) to use for the server."
sensitive = false
}
variable "other_id" {
type = string
description = "The id of the machine image (AMI) to use for the server."
sensitive = true
}

Answer 2

Since the Terraform language makes extensive use of various HCL features that require custom programming with the low-level HCL API, the Terraform team maintains a Go library terraform-config-inspect which understands the Terraform language enough to extract static metadata about top-level objects, including variables. It also deals with the fact that Terraform allows variable definitions in any .tf or .tf.json file interleaved with other declarations; putting them in variables.tf is only a convention.
For example:
mod, diags := tfconfig.LoadModule("examples")
if diags.HasErrors() {
log.Fatalf(diags.Error())
}
for _, variable := range mod.Variables {
fmt.Printf("%#v\n", variable)
}
This library is the same code used by Terraform Registry to produce the documentation about module input variables, so it supports all Terraform language versions that the Terraform Registry does (at the time of writing, going back to the Terraform v0.10 language, since that's the first version that can install modules from a registry) and supports both the HCL native syntax and JSON representations of the Terraform language.