HCL Decoding: Blocks with multiple labels - go

My goal is to parse a HCL configuration (Terraform Configuration) and then write the collected data about variables, outputs, resources blocks and data blocks into a Markdown file.
Variables and outputs are no problem, however, as soon as I trying to decode resource blocks, which have multiple labels.
Works:
variable "foo" {
type = "bar"
}
Doesn't Work:
resource "foo" "bar" {
name = "biz"
}
Error: Extraneous label for resource; Only 1 labels (name) are expected for resource blocks.
Type declaration Code:
import (
"log"
"os"
"strconv"
"github.com/hashicorp/hcl/v2"
"github.com/hashicorp/hcl/v2/gohcl"
"github.com/hashicorp/hcl/v2/hclsyntax"
)
type Variable struct {
Name string `hcl:",label"`
Description string `hcl:"description,optional"`
Sensitive bool `hcl:"sensitive,optional"`
Type *hcl.Attribute `hcl:"type,optional"`
Default *hcl.Attribute `hcl:"default,optional"`
Options hcl.Body `hcl:",remain"`
}
type Output struct {
Name string `hcl:",label"`
Description string `hcl:"description,optional"`
Sensitive bool `hcl:"sensitive,optional"`
Value string `hcl:"value,optional"`
Options hcl.Body `hcl:",remain"`
}
type Resource struct {
Name string `hcl:"name,label"`
Options hcl.Body `hcl:",remain"`
}
type Data struct {
Name string `hcl:"name,label"`
Options hcl.Body `hcl:",remain"`
}
type Config struct {
Outputs []*Output `hcl:"output,block"`
Variables []*Variable `hcl:"variable,block"`
Resources []*Resource `hcl:"resource,block"`
Data []*Data `hcl:"data,block"`
}
Decoding Code:
func createDocs(hclPath string) map[string][]map[string]string {
var variables, outputs []map[string]string
parsedConfig := make(map[string][]map[string]string)
hclConfig := make(map[string][]byte)
c := &Config{}
// Iterate all Terraform files and safe the contents in the hclConfig map
for _, file := range filesInDirectory(hclPath, ".tf") {
fileContent, err := os.ReadFile(hclPath + "/" + file.Name())
if err != nil {
log.Fatal(err)
}
hclConfig[file.Name()] = fileContent
}
// Iterate all file contents
for k, v := range hclConfig {
parsedConfig, diags := hclsyntax.ParseConfig(v, k, hcl.Pos{Line: 1, Column: 1})
if diags.HasErrors() {
log.Fatal(diags)
}
diags = gohcl.DecodeBody(parsedConfig.Body, nil, c)
if diags.HasErrors() {
log.Fatal(diags)
}
}
for _, v := range c.Variables {
var variableType string
var variableDefault string
if v.Type != nil {
variableType = (v.Type.Expr).Variables()[0].RootName()
}
if v.Default != nil {
variableDefault = (v.Default.Expr).Variables()[0].RootName()
}
variables = append(variables, map[string]string{"name": v.Name, "description": v.Description,
"sensitive": strconv.FormatBool(v.Sensitive), "type": variableType, "default": variableDefault})
}
for _, v := range c.Outputs {
outputs = append(outputs, map[string]string{"name": v.Name, "description": v.Description,
"sensitive": strconv.FormatBool(v.Sensitive), "value": v.Value})
}
parsedConfig["variables"], parsedConfig["outputs"] = variables, outputs
return parsedConfig
}
Question: How can I parse multiple labels from resource blocks?

The error you shared is due to the definition of type Resource. resource blocks (and data blocks) in Terraform expect two labels, indicating the resource type and name. To match that in the schema you're implying with these struct types, you'll need to define to fields that are tagged as label:
type Resource struct {
Type string `hcl:"type,label"`
Name string `hcl:"name,label"`
Options hcl.Body `hcl:",remain"`
}
type Data struct {
Type string `hcl:"type,label"`
Name string `hcl:"name,label"`
Options hcl.Body `hcl:",remain"`
}
Although this should work for the limited input you showed here, I want to caution that you are using the higher-level gohcl API which can decode only a subset of HCL that maps well onto Go's struct types. Terraform itself uses the lower-level APIs of hcl.Body and hcl.Expression directly, which allows the Terraform language to include some HCL features that the gohcl API cannot directly represent.
Depending on what your goal is, you may find it better to use the official library terraform-config-inspect, which can parse, decode, and describe a subset of the Terraform language at a higher level of abstraction than the HCL API itself. It also supports modules written for Terraform versions going all the way back to Terraform v0.11, and is the implementation that backs the analysis of modules done by Terraform Registry.

Related

Merging two different structs and marshalling them - Golang

Let's say I have the first struct as
type Person struct {
Name string `json:"person_name"`
Age int `json:"person_age"`
Data map[string]interface{} `json:"data"`
}
and I am trying to marshal an array of the above struct
Things work well till here and a sample response I receive is
[
{
"person_name":"name",
"person_age":12,"data":{}
},
{
"person_name":"name2",
"person_age":12,"data":{}
}
]
Now, I need to append another struct over here and the final response should like
[
{
"person_name":"name",
"person_age":12,"data":{}
},
{
"person_name":"name2",
"person_age":12,"data":{}
},
{
"newData":"value"
}
]
So can someone help on this and how i can achieve this ?
I tried by creating an []interface{} and then iterating over person to append each data, but the issue in this approach is that it makes the Data as null if in case it's an empty string.
I would need it be an empty map only.
Let me prefix this by saying this looks to me very much like you might be dealing with an X-Y problem. I can't really think of many valid use-cases where one would end up with a defined data-type that has to somehow be marshalled alongside a completely different, potentially arbitrary/freeform data structure. It's possible, though, and this is how you could do it:
So you just want to append a completely different struct to the data-set, then marshal it and return the result as JSON? You'll need to create a new slice for that:
personData := []Person{} // person 1 and 2 here
more := map[string]string{ // or some other struct
"newdata": "value",
}
allData := make([]any, 0, len(personData) + 1) // the +1 is for the more, set cap to however many objects you need to marshal
for _, p := range personData {
allData = append(allData, p) // copy over to this slice, because []Person is not compatible with []any
}
allData = append(allData, more)
bJSON, err := json.Marshal(allData)
if err != nil {
// handle
}
fmt.Println(string(bJSON))
Essentially, because you're trying to marshal a slice containing multiple different types, you have to add all objects to a slice of type any (short for interface{}) before marshalling it all in one go
Cleaner approaches
There are much, much cleaner approaches that allow you to unmarshal the data, too, assuming the different data-types involved are known beforehand. Consider using a wrapper type like so:
type Person struct {} // the one you have
type NewData {
NewData string `json:"newdata"`
}
type MixedData struct {
*Person
*NewData
}
In this MixedData type, both Person and NewData are embedded, so MixedData will essentially act as a merged version of all embedded types (fields with the same name should be overridden at this level). With this type, you can marshal and unmarshal the JSON accordingly:
allData := []MixedData{
{
Person: &person1,
},
{
Person: &person2,
},
{
NewData: &newData,
},
}
Similarly, when you have a JSON []byte input, you can unmarshal it same as you would any other type:
data := []MixedData{}
if err := json.Unmarshal(&data, in); err != nil {
// handle
}
fmt.Printf("%#v\n", data) // it'll be there
It pays to add some functions/getters to the MixedData type, though:
func (m MixedData) IsPerson() bool { return m.Person != nil }
func (m MixedData) Person() *Person {
if m.Person == nil {
return nil
}
cpy := *m.Person // create a copy to avoid shared pointers
return &cpy // return pointer to the copy
}
Do the same for all embedded types and this works like a charm.
As mentioned before, should your embedded types contain fields with the same name, then you should override them in the MixedData type. Say you have a Person and Address type, and both have an ID field:
type MixedData struct {
ID string `json:"id"`
*Person
*Address
}
This will set the ID value on the MixedData type, and all other (non-shared) fields on the corresponding embedded struct. You can then use the getters to set the ID where needed, or use a custom unmarshaller, but I'll leave that to you to implement

Generate OpenAPI XML model from Go structure

Is there any way to generate a OpenAPI specification document for XML using golang structure? My structures are annotated with encoding/xml annotations like this:
type Error struct {
Text string `xml:",chardata"`
Type string `xml:"Type,attr"`
Code string `xml:"Code,attr"`
ShortText string `xml:"ShortText,attr"`
}
I would like to generate a OpenAPI model automatically, which respect these annotations
You can use the reflect package from the standard library to inspect the struct and its tags and you can use the gopkg.in/yaml.v3 package to generate the Open API format. You just need to write the logic that translates your type into another, one that matches the structure of the Open API document.
Here's an example of how you could approach this, note that it is incomplete and should therefore not be used as is:
// declare a structure that matches the OpenAPI's yaml document
type DataType struct {
Type string `yaml:"type,omitempty"`
Props map[string]*DataType `yaml:"properties,omitempty"`
XML *XML `yaml:"xml,omitempty"`
}
type XML struct {
Name string `yaml:"name,omitempty"`
Attr bool `yaml:"attribute,omitempty"`
}
// write a marshal func that converts a given type to the structure declared above
// - this example converts only plain structs
func marshalOpenAPI(v interface{}) (interface{}, error) {
rt := reflect.TypeOf(v)
if rt.Kind() == reflect.Struct {
obj := DataType{Type: "object"}
for i := 0; i < rt.NumField(); i++ {
f := rt.Field(i)
tag := strings.Split(f.Tag.Get("xml"), ",")
if len(tag) == 0 || len(tag[0]) == 0 { // no name?
continue
}
if obj.Props == nil {
obj.Props = make(map[string]*DataType)
}
name := tag[0]
obj.Props[name] = &DataType{
Type: goTypeToOpenAPIType(f.Type),
}
if len(tag) > 1 && tag[1] == "attr" {
obj.Props[name].XML = &XML{Attr: true}
}
}
return obj, nil
}
return nil, fmt.Errorf("unsupported type")
}
// have all your types implement the yaml.Marshaler interface by
// delegating to the marshal func implemented above
func (e Error) MarshalYAML() (interface{}, error) {
return marshalOpenAPI(e)
}
// marshal the types
yaml.Marshal(map[string]map[string]interface{}{
"schemas": {
"error": Error{},
// ...
},
})
Try it on playground.

Unable to read terraform variables.tf files into may go program

I am attempting to write a go program that reads in a terraform variables.tf and populates a struct for later manipulation. However, I am getting errors when attempting to "parse" the file. I Am hoping someone can tell me what I am doing wrong:
Code:
package main
import (
"fmt"
"io/ioutil"
"log"
"os"
"github.com/hashicorp/hcl/v2"
"github.com/hashicorp/hcl/v2/gohcl"
"github.com/hashicorp/hcl/v2/hclsyntax"
)
type Config struct {
Upstreams []*TfVariable `hcl:"variable,block"`
}
type TfVariable struct {
Name string `hcl:",label"`
// Default string `hcl:"default,optional"`
Type string `hcl:"type"`
Description string `hcl:"description,attr"`
// validation block
Sensitive bool `hcl:"sensitive,optional"`
}
func main() {
readHCLFile("examples/string.tf")
}
// Exits program by sending error message to standard error and specified error code.
func abort(errorMessage string, exitcode int) {
fmt.Fprintln(os.Stderr, errorMessage)
os.Exit(exitcode)
}
func readHCLFile(filePath string) {
content, err := ioutil.ReadFile(filePath)
if err != nil {
log.Fatal(err)
}
fmt.Printf("File contents: %s", content) // TODO: Remove me
file, diags := hclsyntax.ParseConfig(content, filePath, hcl.Pos{Line: 1, Column: 1})
if diags.HasErrors() {
log.Fatal(fmt.Errorf("ParseConfig: %w", diags))
}
c := &Config{}
diags = gohcl.DecodeBody(file.Body, nil, c)
if diags.HasErrors() {
log.Fatal(fmt.Errorf("DecodeBody: %w", diags))
}
fmt.Println(c) // TODO: Remove me
}
ERROR
File contents: variable "image_id" {
type = string
description = "The id of the machine image (AMI) to use for the server."
sensitive = false
}
variable "other_id" {
type = string
description = "The id of the machine image (AMI) to use for the server."
sensitive = true
}
2021/03/13 19:55:49 DecodeBody: examples/string.tf:2,17-23: Variables not allowed; Variables may not be used here., and 3 other diagnostic(s)
exit status 1
Stack driver question is sadly for hcl1
Blog post I am referencing.
It looks like it's a bug/feature of the library, since as soon as you change string to "string", e.g.,
variable "image_id" {
type = string
...
to
variable "image_id" {
type = "string"
...
gohcl.DecodeBody succeeds.
--- UPDATE ---
So, they do use this package in Terraform, BUT they custom-parse configs, i.e., they don't use gohcl.DecodeBody. They also custom-treat type attributes by using hcl.ExprAsKeyword (compare with description). As you assumed, they do use a custom type for type, but with custom parsing you don't have to.
Below is a working example:
package main
import (
"fmt"
"log"
"os"
"github.com/hashicorp/hcl/v2"
"github.com/hashicorp/hcl/v2/gohcl"
"github.com/hashicorp/hcl/v2/hclsyntax"
)
var (
configFileSchema = &hcl.BodySchema{
Blocks: []hcl.BlockHeaderSchema{
{
Type: "variable",
LabelNames: []string{"name"},
},
},
}
variableBlockSchema = &hcl.BodySchema{
Attributes: []hcl.AttributeSchema{
{
Name: "description",
},
{
Name: "type",
},
{
Name: "sensitive",
},
},
}
)
type Config struct {
Variables []*Variable
}
type Variable struct {
Name string
Description string
Type string
Sensitive bool
}
func main() {
config := configFromFile("examples/string.tf")
for _, v := range config.Variables {
fmt.Printf("%+v\n", v)
}
}
func configFromFile(filePath string) *Config {
content, err := os.ReadFile(filePath) // go 1.16
if err != nil {
log.Fatal(err)
}
file, diags := hclsyntax.ParseConfig(content, filePath, hcl.Pos{Line: 1, Column: 1})
if diags.HasErrors() {
log.Fatal("ParseConfig", diags)
}
bodyCont, diags := file.Body.Content(configFileSchema)
if diags.HasErrors() {
log.Fatal("file content", diags)
}
res := &Config{}
for _, block := range bodyCont.Blocks {
v := &Variable{
Name: block.Labels[0],
}
blockCont, diags := block.Body.Content(variableBlockSchema)
if diags.HasErrors() {
log.Fatal("block content", diags)
}
if attr, exists := blockCont.Attributes["description"]; exists {
diags := gohcl.DecodeExpression(attr.Expr, nil, &v.Description)
if diags.HasErrors() {
log.Fatal("description attr", diags)
}
}
if attr, exists := blockCont.Attributes["sensitive"]; exists {
diags := gohcl.DecodeExpression(attr.Expr, nil, &v.Sensitive)
if diags.HasErrors() {
log.Fatal("sensitive attr", diags)
}
}
if attr, exists := blockCont.Attributes["type"]; exists {
v.Type = hcl.ExprAsKeyword(attr.Expr)
if v.Type == "" {
log.Fatal("type attr", "invalid value")
}
}
res.Variables = append(res.Variables, v)
}
return res
}
Add for completeness, example/string.tf:
variable "image_id" {
type = string
description = "The id of the machine image (AMI) to use for the server."
sensitive = false
}
variable "other_id" {
type = string
description = "The id of the machine image (AMI) to use for the server."
sensitive = true
}
Since the Terraform language makes extensive use of various HCL features that require custom programming with the low-level HCL API, the Terraform team maintains a Go library terraform-config-inspect which understands the Terraform language enough to extract static metadata about top-level objects, including variables. It also deals with the fact that Terraform allows variable definitions in any .tf or .tf.json file interleaved with other declarations; putting them in variables.tf is only a convention.
For example:
mod, diags := tfconfig.LoadModule("examples")
if diags.HasErrors() {
log.Fatalf(diags.Error())
}
for _, variable := range mod.Variables {
fmt.Printf("%#v\n", variable)
}
This library is the same code used by Terraform Registry to produce the documentation about module input variables, so it supports all Terraform language versions that the Terraform Registry does (at the time of writing, going back to the Terraform v0.10 language, since that's the first version that can install modules from a registry) and supports both the HCL native syntax and JSON representations of the Terraform language.

Merge structs with some overlapping fields

I saw several questions asking on how to merge unique structs and how to merge identical structs.
But how would I merge structs that have some overlap? and which fields get taken & when?
e.g.:
type structOne struct {
id string `json:id`
date string `json:date`
desc string `json:desc`
}
and
type structTwo struct {
id string `json:id`
date string `json:date`
name string `json:name`
}
how would I merge it such that I get
{
id string `json:id`
date string `json:date`
desc string `json:desc`
name string `json:name`
}
also, what happens if in this case the two id's are the same (assuming a join over id's) but the names are different?
In javascript, doing something like Object.assign(structOne, structTwo).
Go is a strongly typed language, unlike javascript you can't merge two struct into one combined struct because all type are determined at compile-time. You have two solution here :
Using embedded struct:
One great solution is to use embedded struct because you don't have to merge anything anymore.
package main
import (
"bytes"
"encoding/json"
"fmt"
"log"
)
// Shared field
type common struct {
ID string `json:id`
Date string `json:date`
}
type merged struct {
// Common field is embedded
common
Name string `json:name`
Desc string `json:desc`
}
func main() {
buf := bytes.Buffer{}
buf.WriteString("{ \"id\": \"1\", \"date\": \"27/07/2020\", \"desc\": \"the decription...\" }")
merged := &merged{}
err := json.Unmarshal(buf.Bytes(), merged)
if err != nil {
log.Fatal(err)
}
// Look how you can easily access field from
// embedded struct
fmt.Println("ID:", merged.ID)
fmt.Println("Date:", merged.Date)
fmt.Println("Name:", merged.Name)
fmt.Println("Desc:", merged.Desc)
// Output:
// ID: 1
// Date: 27/07/2020
// Name:
// Desc: the decription...
}
If you want to read more about struct embedding:
golangbyexample.com
travix.io
Using Maps
Another solution is to use maps but you will loose the benefits of struct and methods. This example is not the simplest but there is some great example in the other responses.
In this example I'm using Mergo. Mergo is library that can merge structs and map. Here it is used for creating maps object in the Map methods but you can totally write your own methods.
package main
import (
"fmt"
"log"
"github.com/imdario/mergo"
)
type tOne struct {
ID string
Date string
Desc string
}
// Map build a map object from the struct tOne
func (t1 tOne) Map() map[string]interface{} {
m := make(map[string]interface{}, 3)
if err := mergo.Map(&m, t1); err != nil {
log.Fatal(err)
}
return m
}
type tTwo struct {
ID string
Date string
Name string
}
// Map build a map object from the struct tTwo
func (t2 tTwo) Map() map[string]interface{} {
m := make(map[string]interface{}, 3)
if err := mergo.Map(&m, t2); err != nil {
log.Fatal(err)
}
return m
}
func main() {
dst := tOne{
ID: "destination",
Date: "26/07/2020",
Desc: "destination object",
}.Map()
src := tTwo{
ID: "src",
Date: "26/07/1010",
Name: "source name",
}.Map()
if err := mergo.Merge(&dst, src); err != nil {
log.Fatal(err)
}
fmt.Printf("Destination:\n%+v", dst)
// Output:
// Destination:
// map[date:26/07/2020 desc:destination object iD:destination name:object name
}
Go structs and JavaScript objects are very different. Go structs do not have dynamic fields.
If you want dynamic key/value sets that you can easily iterate over and merge, and are very JSON friendly, why not a map[string]interface{}?
$ go run t.go
map[a:1 b:4]
map[a:1 b:4 c:3]
$ cat t.go
package main
import(
"fmt"
)
type MyObj map[string]interface{}
func (mo MyObj)Merge(omo MyObj){
for k, v := range omo {
mo[k] = v
}
}
func main() {
a := MyObj{"a": 1, "b": 4}
b := MyObj{"b": 2, "c": 3}
b.Merge(a)
fmt.Printf("%+v\n%+v\n", a, b)
}
you can use github.com/fatih/structs to convert your struct to map. Then iterate over that map and choose which fields need to copy over. I have a snippet of code which illustrates this solution.
func MergeStruct (a structOne,b structTwo) map[string]interface{}{
a1:=structs.Map(a)
b1:=structs.Map(b)
/* values of structTwo over writes values of structOne */
var myMap=make(map[string]interface{})
for val,key:=range(a1){
myMap[key]=val
}
for val,key:=range(b1){
myMap[key]=val
}
return myMap
}
You can use the reflect package to do this. Try iterating through the two structs and then you can either use another struct type to store the values or maybe use a map.
Check out this question to find out how you can iterate over a struct.
Check out this question to find out how to get the name of the fields.
Remember to use exported field names for the reflect package to work.
Here is an example which works.

Default value golang struct using `encoding/json` ?

How to set default value for Encoding as "base64"?
type FileData struct {
UID string `json:"uid"`
Size int `json:"size"`
Content string `json:content`
Encoding string `json:encoding`
User string `json:"user"`
}
I tried
Encoding string `json:encoding`= "base64" // Not working
You can set default value when you init your 'FileData'
See my example: https://play.golang.org/p/QXwDG7_mul
Page int has default value 33
package main
import (
"encoding/json"
"fmt"
)
type Response2 struct {
Page int `json:"page"`
Fruits []string `json:"fruits"`
}
func main() {
str := `{"fruits": ["apple", "peach"]}`
res := Response2{Page: 33 /*Default value*/}
json.Unmarshal([]byte(str), &res)
fmt.Println(res)
}
Since your FileData isn't too complex, you can easily make use of json.Unmarshaler interface. Declare Encoding as a separate type and set the default value in the unmarshal method:
type FileData struct {
UID string `json:"uid"`
Size int `json:"size"`
Content string `json:content`
Encoding Encoding `json:encoding` // declared as a custom type
User string `json:"user"`
}
type Encoding string
// implement the Unmarshaler interface on Encoding
func (e *Encoding) UnmarshalJSON(b []byte) error {
var s string
if err := json.Unmarshal(b, &s); err != nil {
return err
}
if s == "" {
*e = Encoding("base64")
} else {
*e = Encoding(s)
}
return nil
}
Now when you encode a json with empty Encoding value, it'll be set to base64:
var data1 = []byte(`{"uid": "UID", "size": 10, "content": "CONTENT", "encoding": "ASCII", "user": "qwe"}`)
var data2 = []byte(`{"uid": "UID", "size": 10, "content": "CONTENT", "encoding": "", "user": "qwe"}`)
func main() {
fmt.Println("Hello, playground")
f := FileData{}
if e := json.Unmarshal(data1, &f); e != nil {
fmt.Println("Error:", e)
}
fmt.Println(f, f.Encoding)
if e := json.Unmarshal(data2, &f); e != nil {
fmt.Println("Error:", e)
}
fmt.Println(f, f.Encoding)
}
Output:
{UID 10 CONTENT ASCII qwe} ASCII
{UID 10 CONTENT base64 qwe} base64
Working code: https://play.golang.org/p/y5_wBgHGJk
You can't because in Go types do not have constructors.
Instead, have either an explicit initializer function (or method on the pointer receiver) or a constructor/factory function (these are conventionally called New<TypeName> so yours would be NewFileData) which would return an initialized value of your type.
All-in-all, I have a feeling this looks like an XY problem. From your question, it appears you want to have a default value on one of your fields if nothing was unmarshaled.
If so, just post-process the values of this type unmarshaled from JSON and if nothing was unmarshaled to Encodning set it to whatever default you want.
Alternatively you might consider this approach:
Have a custom type for your field.
Something like type EncodingMethod string should do.
Have a custom JSON unmarshaling method for this type which would do whatever handling it wants.
If there is a need to read the struct values from environment variables, then "github.com/kelseyhightower/envconfig" library can be used and then you can declare your struct as below:
type FileData struct {
UID string `envconfig:"uid"`
Size int `envconfig:"size"`
Content string `envconfig:content`
Encoding string `envconfig:encoding default:"base64"`
User string `envconfig:"user"`
}
You can finally unmarshal JSON to your struct.
Note: I know this is not what you asked for, but this might help someone.

Resources