parse simple terraform file using go - go

I now tried, everything but cant get this simple thing to work.
I got the following test_file.hcl:
variable "value" {
test = "ok"
}
I want to parse it using the following code:
package hcl
import (
"github.com/hashicorp/hcl/v2/hclsimple"
)
type Config struct {
Variable string `hcl:"test"`
}
func HclToStruct(path string) (*Config, error) {
var config Config
return &config, hclsimple.DecodeFile(path, nil, &config)
But I receive:
test_file.hcl:1,1-1: Missing required argument; The argument "test" is required, but no definition was found., and 1 other diagnostic(s)
I checked with other projects using the same library but I cannot find my mistake .. I just dont know anymore. Can someone guide me into the right direction?

The Go struct type you wrote here doesn't correspond with the shape of the file you want to parse. Notice that the input file has a variable block with a test argument inside it, but the Go type you wrote has only the test argument. For that reason, the HCL parser is expecting to find a file with just the test argument at the top-level, like this:
test = "ok"
To parse this Terraform-like structure with hclsimple will require you to write two struct types: one to represent the top level body containing variable blocks and another to represent the content of each of the blocks. For example:
type Config struct {
Variables []*Variable `hcl:"variable,block"`
}
type Variable struct {
Test *string `hcl:"test"`
}
With that said, I'll note that the Terraform language uses some features of HCL that hclsimple can't support, or at least can't support with direct decoding like this. For example, the type argument inside variable blocks is a special sort of expression called a type constraint which hclsimple doesn't support directly, and so parsing it will require using the lower-level HCL API. (That's what Terraform is doing internally.)

Thanks to #Martin Atkins, I now have the following code that works:
type Config struct {
Variable []Variable `hcl:"variable,block"`
}
type Variable struct {
Value string `hcl:"name,label"`
Test string `hcl:"test"`
}
With that I can parse a variables.tf like:
variable "projectname" {
default = "cicd-template"
}
variable "ansibleuser" {
default = "centos"
}

Related

How to represent golang "any" type in protobuf language

am using a protobuf definition file to create golang classes
file.proto
message my_message {
<what type to use?> data = 1 [json_name = "data"]
}
generated.go
type MyMessage struct {
data any
}
I have checked out Struct , Value and Any, though none of them actually mark the type as “any” in generated classes.
The closes I get is Value type which can accomodate primitives as well as structs and lists.
As far as I'm aware the Go code generator doesn't have any circumstances where any/interface{} will be used for a generated field.
google.protobuf.Any is likely what you want here, but the field in the generated code will of type anypb.Any. If you absolutely require the any type be used, you'll unfortunately have to manually map to another struct.

How to handle NaN values when writing to parquet in GO?

I am trying to write to a parquet file in GO. While writing to this file, I can get NaN values. Since NaN is neither defined in the primitive types nor in logical type then how do I handle this value in GO? Does any existing schema work for it?
I am using the parquet GO library from here. You can find an example of the code using JSON schema for writing to parquet here using this library.
The isse was discussed at lenght in xitongsys/parquet-go issue 281, with the recommandation being to
use OPTIONAL type.
Even you don't assign a value (like you code), the non-point value will be assigned a default value.
So parquet-go don't know it's null or default value.
However:
What is comes down to is that I cannot use the OPTIONAL type, in other words I cannot convert my structure to use pointers.
I have tried to use repetitiontype=OPTIONAL as a tag, but this leads to some weird behavior.
I would expect that tag to behave the same way that the omitempty tag in the Golang standard library, i.e. if the value is not present then it is not put into the JSON.
The reason this is important is that if the field is missing or not set, when it is encoded to parquet then there is no way of telling if the value was 0 or just not set in the case of int64.
This illustrates the issue:
package main
import (
"encoding/json"
"io/ioutil"
)
type Salary struct {
Basic, HRA, TA float64 `json:",omitempty"`
}
type Employee struct {
FirstName, LastName, Email string `json:",omitempty"`
Age int
MonthlySalary []Salary `json:",omitempty"`
}
func main() {
data := Employee{
Email: "mark#gmail.com",
MonthlySalary: []Salary{
{
Basic: 15000.00,
},
},
}
file, _ := json.MarshalIndent(data, "", " ")
_ = ioutil.WriteFile("test.json", file, 0o644)
}
with a JSON produced as:
{
"Email": "mark#gmail.com",
"Age": 0,
"MonthlySalary": [
{
"Basic": 15000
}
]
}
As you can see, the item in the struct that have the omit empty tag and that are not assigned do no appear in the JSON, i.e. HRA TA.
But on the other hand Age does not have this tag and hence it is still included in the JSON.
This is problematic as all fields in the struct are assigned memory when this golang library writes to parquet- so if you have a big struct that is only sparsely populated it will still take the full amount of memory.
It is a bigger problem when the file is read again as there is no way of know if the value that was put in the parquet file was the empty value or it is was just not assigned.
I am happy to help implement an omitempty tag for this library if I can convince you of the value of having it.
That echoes issue 403 "No option to omitempty when not using pointers".

Go Protobuf declarations and optional Fields in Go Struct (string pointers)

I am running into a bit of a problem with Protoc and my existing struct that contains nullable string fields.
The struct I am trying to serialize for transmission contains a bunch of fields that are nullable in json (so we can distinguish between null, "" and a set value).
type Message struct {
Path *string `json:"path"`
}
So if a user sends a empty json string {} the Path will be nil and not "", whereas {"path":""} is also valid and a different case from {"path": null}.
The proto3 declaration I came up with obviously looks like this (and is optional as required and optional got dropped from proto3:
syntax = "proto3";
message Message {
string Path = 1;
}
After running Protoc I end up with a struct that looks like this and all values are string and no way to declare them as *string:
type Message struct {
Path string `protobuf:"bytes,1,opt,name=Path,proto3" json:"Path,omitempty"`
}
Obviously I can't assign to this array from my existing struct. But even if I were to write the tedious mapping code with target.Path = *source.Path with the appropriate null pointer checks etc I'd loose the triple meaning of my source struct (nil, "", "value").
Any suggestions on how to proceed here or if there is an extension to the Go Protobuf to do this? Or how would one go about describing this proto declaration?
Proto3 returns the Zero Value even if a field isn't set. Currently there is no way to distinguish if a field has been set or not.
See Github issue #15.
Possible solutions:
Use proto2 instead of proto3.
Use gogoproto's nullable extension.
Use the google.protobuf.FieldMask extension, see Common Design Patterns
from the Google API Design Guide: Partial Responses and Output Fields.
https://stackoverflow.com/a/62566052
proto3 support optional with flag --experimental_allow_proto3_optional
In my case, I solved this problem with several packages:
https://github.com/gogo/protobuf
https://github.com/golang/protobuf
My proto file looks like this:
syntax = "proto3";
import "google/protobuf/wrappers.proto";
import "github.com/gogo/protobuf/gogoproto/gogo.proto";
message Message {
google.protobuf.StringValue path = 1 [(gogoproto.wktpointer) = true];
}
The command for generating the go code, which I used look like this:
protoc -I. -I%GOPATH%/src --gogofaster_out=plugins=grpc:. proto/*.proto
The generated go file looks like this:
type Message struct {
Path *string `protobuf:"bytes,1,opt,name=path,json=path,proto3,wktptr" json:"path,omitempty"`
}

Embedded structs get all tags and set values for all fields based on tag

I'm designing config component where I want to set ENV variable for each field on go struct.
e.g.
type SubStruct struct {
Field int `env:"ENV_INT"`
}
type Config struct {
Name string `env:"APP_NAME"`
Sub1 struct {
Sub11 struct {
Value string `env:"ENV_VAR"`
}
Sub12 SubStruct
}
}
What I want to achieve is to:
Get values of all env keys recursively
Set value based on field type
So there are basically two questions:
Is there any lib in go which exposes methods for setting values using reflection (e.g. like private methods in json package)?
Is there any method/function in go which can parse your struct recursively and get all tags from struct?
edit:
- updated structs
- renamed confusing field names (from embed to sub)

Go vet: "composite literal uses unkeyed fields" with embedded types

I have a simple structure:
type MyWriter struct {
io.Writer
}
Which I then use in the following way:
writer = MyWriter{io.Stdout}
When running go vet this gives me a composite literal uses unkeyed fields.
In order to fix this would I have to turn io.Reader into a field in the MyWriter structure by adding a key?
type MyWriter struct {
w io.Writer
}
Is there any other way around this?
The only other answer I found on here suggests to disable the check altogether but I would rather not do that and find a proper solution.
Try this:
writer = MyWriter{Writer: io.Stdout}
Embedded structs have an implicit key of the type name itself without the package prefix (e.g. in this case, Writer).

Resources