I'm new in go, I can't find a way to unmarshal a yaml using "gopkg.in/yaml.v2"
I suppose the error is in the way I define the struct.
I need to parse a kubernetes job yaml and edit in go to generate an update yaml.
The structure is almost static but have two lists in which the keys could have different things inside.
I reduced the yaml to one listes (volumes) to simplify the example.
apiVersion: batch/v1
kind: Job
metadata:
name: jobname
namespace: namespace
spec:
ttlSecondsAfterFinished: 86400
template:
spec:
containers:
- name: container-name
image: containerimage:tag
command:
- php
- cli/migrations.php
- up
restartPolicy: Never
volumes:
- name: filestore
persistentVolumeClaim:
claimName: data-pvc
readOnly: false
- name: stackdriver
secret:
secretName: stackdriver-prod
backoffLimit: 1
those are my structs definitions:
type PersistentVolumeClaims struct {
ClaimName string `yaml:"claimName,omitempty"`
ReadOnly bool `yaml:"readOnly,omitempty"`
}
type Secrets struct {
SecretName string `yaml:"secretName,omitempty"`
}
type Names struct {
Name string `yaml:"name"`
PersistentVolumeClaim PersistentVolumeClaims `yaml:"persistentVolumeClaim,omitempty"`
Secret Secrets `yaml:"secret,omitempty"`
}
type Jobs struct {
ApiVersion string `yaml:"apiVersion"`
Kind string `yaml:"kind"`
Metadata struct {
Name string `yaml:"name"`
Namespace string `yaml:"namespace"`
}
Spec struct {
TtlSecondsAfterFinished int `yaml:"ttlSecondsAfterFinished"`
Template struct {
Spec struct {
Containers []struct {
Name string
Image string `yaml:"image"`
Command []string `yaml:"command"`
VolumeMounts []struct {
Name string
SubPath string `yaml:"subPath"`
MountPath string `yaml:"mountPath"`
ReadOnly bool `yaml:"readOnly"`
}
RestartPolicy string `yaml:"restartPolicy"`
}
Volumes map[string][]Names
}
BackoffLimit int `yaml:"backoffLimit"`
}
}
}
I tried different structures but I don't get the solution.
Any help will be appreciated.
--- SOLVED
I have redone the tool using the official go-client https://github.com/kubernetes/client-go as suggested by Jonas. Now everything work!
I need to parse a kubernetes job yaml and edit in go to generate an update yaml. The structure is almost static but have two lists in wich the keys could have different things inside.
It sound like your application is running in a cluster, want to retrieve a Job, modify it and then update it.
I would recommend the official Kubernetes client-go for this. It has libraries for Kubernetes resources like Job. See the example using a Deployment
You can use the official structs from the Kubernetes APIs. If you still want to declare the structs: Your yaml tags are incomplete. You do not have tags for nested structs:
type Jobs struct {
Metadata struct {
Name string `yaml:"name"`
Namespace string `yaml:"namespace"`
} `yaml:"metadata"` // Here is one
Spec struct {
TtlSecondsAfterFinished int `yaml:"ttlSecondsAfterFinished"`
Template struct {
Spec struct {
Containers []struct {
VolumeMounts []struct {
Name string
SubPath string `yaml:"subPath"`
MountPath string `yaml:"mountPath"`
ReadOnly bool `yaml:"readOnly"`
} `yaml:"volumeMounts"` // Here is another
} `yaml:"containers"` // Here is another
Volumes map[string][]Names `yaml:"volumes"` // This is missin as well
} `yaml:"spec"` //
} `yaml:"template"` //
} `yaml:"spec"` //
}
Suppose that your data is in the file example.yaml; you can unmarshall into a struct using gopkg.in/yaml.v2 like
package main
import (
"fmt"
"log"
"io/ioutil"
"gopkg.in/yaml.v2"
)
type item struct {
ItemA string `yaml:"item_a"`
ItemB string `yaml:"item_b"`
}
func read(filename string) (item, error) {
var output item
content, err := ioutil.ReadFile(filename)
if err != nil {
return item{}, err
}
err = yaml.Unmarshal(content, output)
if err != nil {
return item{}, err
}
return output, nil
}
func main() {
output, err := read("example.yaml")
if err != nil {
log.Fatal(err)
}
fmt.Println("output: ", output)
}
Related
The YAML file:
namespaces:
- namespace: default
aliasname: k8s
components:
- component: comp1
replicas: 1
port: 8080
- component: comp2
replicas: 1
port: 9999
- namespace: ns2
components:
- component: comp1
replicas: 1
From the YAML file above, I want to create structs like the following:
type Namespaces struct {
NamespaceName string `yaml:"namespace"`
Aliasname string `yaml:"aliasname,omitempty"`
ListOfComponents []Components `yaml:"components"`
ComponentMap map[string]Components
}
type Components struct {
ComponentName string `yaml:"component"`
NumReplicas int `yaml:"replicas"`
Port int `yaml:"port"`
}
type Config struct {
ListOfNamespaces []Namespaces `yaml:"namespaces"`
NamespaceMap map[string]Namespaces
}
The fields Namespacemap and Componentmap should be able to be retrieved when accessing the config and namespace object respectively. I created a method to convert the list of the namespaces and components into maps, but when I call config.Namespacemap or Namespace.ComponentMap, it returns an empty map.
Basically I would like to know: How do we add extra fields to type structs? I would like to access new variables like a map from the config struct.
Update:
Thanks blami for guiding me, but when I try to write the same for Components, it doesn't give me the whole namespacemap with the componentmap included:
type Components struct {
ComponentName string `yaml:"component"`
NumReplicas int `yaml:"replicas"`
Port int `yaml:"port"`
}
type Namespaces struct {
NamespaceName string `yaml:"namespace"`
Aliasname string `yaml:"aliasname"`
ComponentMap map[string]Components `yaml:"components"`
}
func (n *Namespaces) UnmarshalYAML(unmarshal func(interface{}) error) error {
type origNamespace struct {
ListOfComponents []Components `yaml:"components"`
}
var on origNamespace
err1 := unmarshal(&on)
if err1 != nil {
return err1
}
n.ComponentMap = make(map[string]Components)
for _, i := range on.ListOfComponents {
n.ComponentMap[i.ComponentName] = i
}
return nil
}
When I run the config.NamespaceMap it gives the following
map[:{NamespaceName: K8sNamespace: ComponentMap:map[comp1:{ComponentName:comp1 NumShards:0 NumReplicas:1 Port:0 EpochLength:0}]}]
If you want to do such transformation you will need to write a customized UnmarshalYAML() receiver on types Config and Namespace. Here is basic working example of doing so (only for "namespaces"):
type Config struct {
Namespaces map[string]Namespace `yaml:"namespaces"`
}
// Replace go-yaml built-in unmarshaller with custom that will transform list to map.
// Note this code is meant only as demonstration!
func (c *Config) UnmarshalYAML(unmarshal func(interface{}) error) error {
type origConfig struct {
Namespaces []Namespace `yaml:"namespaces"`
}
var o origConfig
err := unmarshal(&o)
if err != nil {
return err
}
// Assign namespaces in list to map; namespaces with same name will be overwritten
c.Namespaces = make(map[string]Namespace)
for _, n := range o.Namespaces {
c.Namespaces[n.Namespace] = n
}
return nil
}
// You can use "Config" type as usual
func main() {
var config Config
err := yaml.Unmarshal(<your_yaml>, &config)
if err != nil {
log.Panic(err)
}
fmt.Println(config.Namespaces)
// map[default:{default k8s} ns2:{ns2 }]
fmt.Printf("%v\n", config.Namespaces["default"])
// {default k8s}
}
As noted in code example this will cause some problems (e.g. what to do if namespace names are same?).
Playground link: https://go.dev/play/p/IKg8kmRnknq
I have a problem unmarshaling a simple slice of YAML data:
package main
import (
"fmt"
"gopkg.in/yaml.v2"
)
type myDataStruct struct {
HTTP []struct {
Name string
Target string
}
}
func main() {
yamlData := `
HTTP:
- name: one
target: http://wazaa
- name: two
target: http://wazii
`
var myData myDataStruct
err := yaml.Unmarshal([]byte(yamlData), &myData)
if err != nil {
fmt.Printf("error: %v", err)
return
}
fmt.Print(myData)
}
Playground: https://play.golang.org/p/Srb2DJVVZqN
The result is {[]} and my limited knowledge of Go does not help to understand why?
If you don't specify the mapping between labels used in the YAML source and Go struct fields, by default they will only be matched if only changing the first letter to lower matches.
E.g. the struct field Name will match name, but not NAME.
Specify the mapping for the HTTP field:
type myDataStruct struct {
HTTP []struct {
Name string
Target string
} `yaml:"HTTP"`
}
With this change it works and outputs (try it on the Go Playground):
{[{one http://wazaa} {two http://wazii}]}
It's good practice to provide mappings for all fields, so it'll continue to work if you rename the fields:
type myDataStruct struct {
HTTP []struct {
Name string `yaml:"name"`
Target string `yaml:"target"`
} `yaml:"HTTP"`
}
Although in your case the default matching works for the Name and Target fields without providing the mappings.
I am trying to unmasrhal the following flux HelmRelease file.
apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
annotations:
fluxcd.io/automated: 'false'
fluxcd.io/tag.ats: glob:*
name: ats
namespace: myns
spec:
chart:
git: git#github.com:reponame/project.git
path: charts/path1/path1/myapp
ref: master
releaseName: foobar
values:
allowAllEgress: true
recycleApp: true
hooks:
slackChannel: https://hooks.slack.com/services/something/somethingelse/
Here are my models
type HelmReleaseValues struct {
AllowAllEgress bool `yaml:"allowAllEgress"`
RecycleApp bool `yaml:"recycleApp"`
Hooks `yaml:"hooks"`
}
type Hooks struct {
SlackChannel string `yaml:"slackChannel"`
}
type Values struct {
HelmReleaseValues `yaml:"values"`
ReleaseName string `yaml:"releaseName"`
Chart `yaml:"chart"`
}
type Spec struct {
Values `yaml:"spec"`
}
The problem is that the fields allowAllEgress and recycleApp are getting unmarshalled.
However the Hooks field in my struct turns out to be empty.
What am I doing wrong in the struct modelling / tagging?
edit: here is my code
package main
import (
"fmt"
"io/ioutil"
"os"
"github.com/davecgh/go-spew/spew"
"gopkg.in/yaml.v3"
)
const ExitCodeCmdErr = 1
func main() {
rawYaml := parseHelmReleaseFile("myfile.yaml")
spew.Dump(rawYaml)
}
func parseHelmReleaseFile(fileName string) Spec {
var v Spec
yamlFile, err := ioutil.ReadFile(fileName)
if err != nil {
fmt.Printf("yaml file err #%v ", err)
os.Exit(ExitCodeCmdErr)
}
err = yaml.Unmarshal(yamlFile, &v)
if err != nil {
fmt.Printf("Unmarshal: %v", err)
os.Exit(ExitCodeCmdErr)
}
return v
}
I am running the program and grepping for the output (the actual helm release file is huge)
▶ go clean && gb .
~/Desktop/yamltutorial
./foobar | grep -i hooks -A 3
--
Hooks: (main.Hooks) {
SlackChannel: (string) ""
}
},
You did not have Chart struct
type Chart struct {
Git string `yaml:"git"`
Path string `yaml:"path"`
Ref string `yaml:"ref"`
}
Added that and got the following output
{Values:{HelmReleaseValues:{AllowAllEgress:true RecycleApp:true Hooks:{SlackChannel:https://hooks.slack.com/services/something/somethingelse/}} ReleaseName:foobar Chart:{Git:git#github.com:reponame/project.git Path:charts/path1/path1/myapp Ref:master}}}
Playground file with complete code.
https://play.golang.org/p/vCnjApr6gI9
I want to unmarshal my ~/.kube/config file into a go struct.
I am using the following approach
func ListContexts(pathToKubeConfig string) ([]string, error) {
type Contexts struct {
Ctx []string `yaml:"contexts"`
//ApiVersion string `yaml:"apiVersion"`
}
var ctx []string
var c Contexts
file, err := ioutil.ReadFile(pathToKubeConfig)
if err != nil {
return nil, err
}
err = yaml.Unmarshal(file, &c)
fmt.Printf("%#v\n", c.Ctx)
return ctx, nil
}
}
As is widely known, a kubeconfig file has the following struct:
apiVersion: v1
. . .
contexts:
- context:
cluster: cluster1
user: user1
name: context1
- context:
cluster: cluster2
user: user2
name: context2
My approach is printing:
[]string(nil)
Since context is a yaml array, why my mapping into a string array not working?
When I uncomment the ApiVersion field of my struct and try to print it, it works.
context is an array, but not a string array. Either use []map[string]interface{} for context, or define the context as a struct, and use its array:
type context struct {
Cluster string `yaml:"cluster"`
...
}
type contexts struct {
Contexts []context `yaml:"contexts"`
}
I need to grab some pod information which will be used for some unit tests which will be run in-cluster. I need all the information which kubectl describe po gives but from an in cluster api call.
I have some working code which makes an api call to apis/metrics.k8s.io/v1beta1/pods, and have installed the metrics-server on minikube for testing which is all working and gives me output like this:
Namespace: kube-system
Pod name: heapster-rgnlj
SelfLink: /apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/heapster-rgnlj
CreationTimestamp: 2019-09-10 12:27:13 +0000 UTC
Window: 30s
Timestamp: 2019-09-10 12:26:23 +0000 UTC
Name: heapster
Cpu usage: 82166n
Mem usage: 19420Ki
...
func getMetrics(clientset *kubernetes.Clientset, pods *PodMetricsList) error {
data, err := clientset.RESTClient().Get().AbsPath("apis/metrics.k8s.io/v1beta1/pods").DoRaw()
if err != nil {
return err
}
err = json.Unmarshal(data, &pods)
return err
}
func main() {
config, err := rest.InClusterConfig()
if err != nil {
fmt.Println(err)
}
// creates the clientset
clientset, err := kubernetes.NewForConfig(config)
if err != nil {
fmt.Println(err)
}
var pods PodMetricsList
err = getMetrics(clientset, &pods)
if err != nil {
fmt.Println(err)
}
for _, m := range pods.Items {
fmt.Print("Namespace: ", m.Metadata.Namespace, "\n", "Pod name: ", m.Metadata.Name, "\n", )
fmt.Print("SelfLink: ", m.Metadata.SelfLink, "\n", "CreationTimestamp: ", m.Metadata.CreationTimestamp, "\n", )
fmt.Print("Window: ", m.Window, "\n", "Timestamp: ", m.Timestamp, "\n", )
for _, c := range m.Containers {
fmt.Println("Name:", c.Name)
fmt.Println("Cpu usage:", c.Usage.CPU)
fmt.Println("Mem usage:", c.Usage.Memory, "\n")
...
As I say, what i really need is what you'd get with a 'describe pods' type call. Having looked through the kubernetes source this NodeDescriber looks like the right type of function, but I'm slightly at a loss as to how to integrate / implement it to get the desired results.
kubernetes/pkg/printers/internalversion/describe.go
Line 2451 in 4f2d7b9
func (d *NodeDescriber) Describe(namespace, name string, describerSettings...etc)
I'm new to Go and not particularly familiar with kubernetes.
Any pointers as to how to go about it would be greatly appreciated.
Looking at the describePod and Describe funcs from staging/src/k8s.io/kubectl/pkg/describe/versioned/describe.go should give you a better picture of how to do this. And since Describe and PodDescriber are public, you can reuse these for your use case.
You could couple this with a CoreV1Client which has a Pods func, that returns a PodInterface that has a List func which would return a list of Pod objects for the given namespace.
Those pod objects will provide the Name needed for the Describe func, the Namespace is already known, and the describe.DescriberSettings is just a struct type that you could inline to enable showing events in the Describe output.
Using the List func will only list the pods that one time. If you're interested in having this list be updated regularly, you might want to look at the Reflector and Informer patterns; both of which are largely implemented in the tools/cache package, and the docs briefly explain this concept in the Efficient detection of changes section.
Hope this helps.
I didn't try, but I would suggest to start with:
1. using kubectl with --verbosity option to see the full api request
kubectl describe pod xxx -v=8
like:
GET https://xx.xx.xx.xx:6443/api/v1/namespaces/default/events?fieldSelector=involvedObject.uid%3Ddd77c4aa-28e6-4bf0-8dfe-0d8610cbe9c9%2CinvolvedObject.name%3Dmy-app%2CinvolvedObject.namespace%3Ddefault
It contains fields related to your POD: fieldSelector=involvedObject.uid, involvedObject.name, involvedObject.namespace
2. I thing the good start it will be the code from github func describePod to start with.
Hope this help.
Well I ended up just writing out a struct to map the results from a /api/v1/pods query and just iterating through ranges to get what I need.
Here's the struct in case it saves anybody else some time.
type ApiV1PodMetricsList struct {
Kind string `json:"kind"`
APIVersion string `json:"apiVersion"`
Metadata struct {
SelfLink string `json:"selfLink"`
ResourceVersion string `json:"resourceVersion"`
} `json:"metadata"`
Items []struct {
Metadata struct {
Name string `json:"name"`
Namespace string `json:"namespace"`
SelfLink string `json:"selfLink"`
UID string `json:"uid"`
ResourceVersion string `json:"resourceVersion"`
CreationTimestamp time.Time `json:"creationTimestamp"`
Labels struct {
Run string `json:"run"`
} `json:"labels"`
} `json:"metadata"`
Spec struct {
Volumes []struct {
Name string `json:"name"`
Secret struct {
SecretName string `json:"secretName"`
DefaultMode string `json:"defaultMode"`
} `json:"secret"`
} `json:"volumes"`
Containers []struct {
Name string `json:"name"`
Image string `json:"image"`
Resources struct {
} `json:"resources"`
VolumeMounts []struct {
Name string `json:"name"`
ReadOnly string `json:"readOnly"`
MountPath string `json:"mountPath"`
} `json:"volumeMounts"`
TerminationMessagePath string `json:"terminationMessagePath"`
TerminationMessagePolicy string `json:"terminationMessagePolicy"`
ImagePullPolicy string `json:"imagePullPolicy"`
Stdin string `json:"stdin"`
StdinOnce string `json:"stdinOnce"`
Tty string `json:"tty"`
} `json:"containers"`
RestartPolicy string `json:"restartPolicy"`
TerminationGracePeriodSeconds string `json:"terminationGracePeriodSeconds"`
DnsPolicy string `json:"dnsPolicy"`
ServiceAccountName string `json:"serviceAccountName"`
ServiceAccount string `json:"serviceAccount"`
NodeName string `json:"nodeName"`
SecurityContext struct {
} `json:"securityContext"`
SchedulerName string `json:"schedulerName"`
Tolerations []struct {
Key string `json:"key"`
Operator string `json:"operator"`
Effect string `json:"effect"`
TolerationSeconds string `json:"tolerationSeconds"`
} `json:"tolerations"`
Priority string `json:"priority"`
EnableServiceLinks string `json:"enableServiceLinks"`
} `json:"spec"`
Status struct {
Phase string `json:"phase"`
Conditions []struct {
Type string `json:"type"`
Status string `json:"status"`
LastProbeTime time.Time `json:"lastProbeTime"`
LastTransitionTime time.Time `json:"lastTransitionTime"`
Reason string `json:"reason"`
} `json:"conditions"`
HostIP string `json:"hostIP"`
PodIP string `json:"podIP"`
StartTime string `json:"startTime"`
ContainerStatuses []struct {
Name string `json:"name"`
State struct {
Terminated struct {
ExitCode string `json:"exitCode"`
Reason string `json:"reason"`
StartedAt time.Time `json:"startedAt"`
FinishedAt time.Time `json:"finishedAt"`
ContainerID string `json:"containerID"`
} `json:"terminated"`
} `json:"state"`
LastState struct {
} `json:"lastState"`
Ready bool `json:"ready"`
RestartCount int64 `json:"restartCount"`
Image string `json:"image"`
ImageID string `json:"imageID"`
ContainerID string `json:"containerID"`
} `json:"containerStatuses"`
QosClass string `json:"qosClass"`
} `json:"status"`
} `json:"items"`
}