I have created a cluster on Alibaba.
I need to fetch cluster data in the Golang project.
Getting below error from API:
{
"Code": 403,
"Message": "namespaces is forbidden: User \"281247226166595041\" cannot list resource \"namespaces\" in API group \"\" at the cluster scope"
}
Tried accessing it via kubectl:
$ kubectl get namespace
Error from server (Forbidden): namespaces is forbidden: User "225396037912844073" cannot list resource "namespaces" in API group "" at the cluster scope
Not able to fetch data for cluster created by another user.
Please help me with this issue.
That's issue of authentication
Your user doesn't have access to list that namespace.
You need to update the RBAC or user access, or however, you are authenticating the Go client into Kubernetes.
If you are using the service account to give access check the RBAC.
Give service account cluster-admin access
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: <new-account-name>
namespace: <namespace>
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: go-rbac
subjects:
- kind: ServiceAccount
name: default
namespace: default
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
kubectl apply -f .yaml
If you are running outside the cluster make sure your script pointing to correct kubeconfig
package main
import (
"fmt"
"k8s.io/client-go/1.5/kubernetes"
"k8s.io/client-go/1.5/pkg/api/v1"
"k8s.io/client-go/1.5/tools/clientcmd"
)
func main() {
config, err := clientcmd.BuildConfigFromFlags("", <kube-config-path>)
if err != nil {
return nil, err
}
c, err := kubernetes.NewForConfig(config)
if err != nil {
return nil, err
}
Running inside the custer
package main
import (
"context"
"fmt"
"time"
"k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/rest"
//
// Uncomment to load all auth plugins
// _ "k8s.io/client-go/plugin/pkg/client/auth"
//
// Or uncomment to load specific auth plugins
// _ "k8s.io/client-go/plugin/pkg/client/auth/azure"
// _ "k8s.io/client-go/plugin/pkg/client/auth/gcp"
// _ "k8s.io/client-go/plugin/pkg/client/auth/oidc"
// _ "k8s.io/client-go/plugin/pkg/client/auth/openstack"
)
func main() {
// creates the in-cluster config
config, err := rest.InClusterConfig()
if err != nil {
panic(err.Error())
}
// creates the clientset
clientset, err := kubernetes.NewForConfig(config)
if err != nil {
panic(err.Error())
}
Example : https://github.com/kubernetes/client-go/blob/master/examples/in-cluster-client-configuration/main.go
Related
When I get the kubernetes resources from api with client-go, but I can't found the apiversion and kind in the response, the apiversion and kind is empty. How can I get the apiversion and kind of the resource?
below is my code:
package main
import (
"flag"
"k8s.io/client-go/tools/clientcmd"
"log"
"k8s.io/client-go/kubernetes"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"fmt"
)
var clientset *kubernetes.Clientset
func main() {
k8sconfig := flag.String("k8sconfig","./k8sconfig","kubernetes config file path")
flag.Parse()
config , err := clientcmd.BuildConfigFromFlags("",*k8sconfig)
if err != nil {
log.Println(err)
}
clientset , err = kubernetes.NewForConfig(config)
if err != nil {
log.Fatalln(err)
} else {
fmt.Println("connect k8s success")
}
pods,err := clientset.CoreV1().Pods("").List(metav1.ListOptions{})
if err != nil {
log.Println(err.Error())
}
for _, pod := range pods.Items{
fmt.Println("apiversion: ", pod.APIVersion, "kind: ", pod.Kind)
}
}
The output:
apiversion: kind:
apiversion: kind:
apiversion: kind:
apiversion: kind:
apiversion: kind:
apiversion: kind:
apiversion: kind:
apiversion: kind:
......
......
I think the issue is that you are getting the list of pods using the List() API so it is not a pod as you expect and doesn't have a Kind field.
You need to iterate over the list of pods to access individual pods:
for _, pod := range pods.Items {
fmt.Printf("%s %s\n", pod.GetName(), pod.GetCreationTimestamp())
}
The Kind field is present as part of Pod's Metadata and can be accessed using pod.ObjectMeta.Kind.
You are not getting the APIVersion and Kind because it was ignored from the code.
If you take a look at the API server JSON response, it will be something like below:
{
"kind":"PodList",
"apiVersion":"v1",
"metadata":{
"resourceVersion":"2397"
},
"items":[
{
... ... ...
So the response does contain the APIVersion and Kind. But when the response is decoded to k8s object, here
out, _, err := r.decoder.Decode(r.body, nil, obj)
Here you can see that the second output parameter is ignored, which is schema.GroupVersionKind.
func (c *Something) Decode([]byte, *schema.GroupVersionKind, runtime.Object) (runtime.Object, *schema.GroupVersionKind, error) {
}
N.B.: When you are making an API call using the client-go (unless dynamic client), you already know the APIVersion (ie. CoreV1() ) and the Kind ( ie. List()).
I want to create a custom kubernetes resource with go. The application is deployed in the kubernetes cluster. I want to create e.g. the followng resource:
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: add-response-header
config:
add:
headers:
- "demo: injected-by-kong"
plugin: response-transformer
So far I always created the 'standard' resources e.g. a secret with the following code:
CreateSecret(name string, data map[string]string) error {
confs, err := rest.InClusterConfig()
if err != nil {
panic(err)
}
clientset, err = kubernetes.NewForConfig(confs)
i := clientset.CoreV1()
if _, err := i.Secrets(namespace).Create(&v1.Secret{
TypeMeta: metav1.TypeMeta{
Kind: "Secret",
APIVersion: "v1",
},
ObjectMeta: metav1.ObjectMeta{
Name: name,
},
StringData: data,
Type: "Opaque",
}); err != nil {
return err
}
}
In addition I tried to get a resource with the following code:
b, err := clientset.RESTClient().Get().Namespace(namespace).Resource("KongPlugin").DoRaw()
I get the following err:
the server could not find the requested resource (get KongPlugin)
If I make a request at the command line k get KongPlugin I can see all the resources.
NAME PLUGIN-TYPE AGE
add-proxy-headers request-transformer 3h34m
So how can view the custom resoources?
For RESTClient ๐๐ผ
Get:
You have to fully specify path to you custom resource.
Use fluent interface
data, err := clientset.RESTClient().
Get().
AbsPath("/apis/<api>/<version>").
Namespace("<namespace>").
Resource("kongplugins").
Name("kongplugin-sample").
DoRaw(context.TODO())
or specify manually
data, err := clientset.RESTClient().
Get().
AbsPath("/apis/<api>/<version>/namespaces/<namespace>/kongplugins/kongplugin-sample").
DoRaw(context.TODO())
You can find AbsPath in selfLink of custom resource.
Create:
For example, you can post marshaled data use AbsPath
kongPlugin := &KongPlugin{
TypeMeta: metav1.TypeMeta{
APIVersion: "<api>/<version>",
Kind: "KongPlugin",
},
ObjectMeta: metav1.ObjectMeta{
Name: "kongplugin-sample",
Namespace: "<namespace>",
},
...}}
body, err := json.Marshal(kongPlugin)
data, err := clientset.RESTClient().
Post().
AbsPath("/apis/<api>/<version>/namespaces/<namespace>/kongplugins").
Body(body).
DoRaw(context.TODO())
because arg of the method Body(obj interface{}) is an empty interface, you can use different types of arguments according to documentation:
k8s.io/client-go/rest - func (*Request) Body
Is there documentation on building k8s jobs in go-client somewhere? In particular I'm trying to convert a job yaml to go code and for the life of me cannot find reference docs that say how the fields convert
k8s.io/api is a package of Kubernetes which kubectl and other components use it to implement Kubernetes APIs. In this package, there is a struct that implements the Job API and you can use it to convert Job manifests to go structs.
I think this code can help:
package main
import (
"fmt"
"io/ioutil"
"os"
"gopkg.in/yaml.v2"
v1 "k8s.io/api/batch/v1"
)
func main() {
file, err := os.Open("/path/to/job.yaml")
if err != nil {
panic(err)
}
b, err := ioutil.ReadAll(file)
if err != nil {
panic(err)
}
job := &v1.Job{}
err = yaml.Unmarshal(b, job)
if err != nil {
panic(err)
}
fmt.Println(job)
}
Converting YAML to Golang can be difficult, and often times the documentation with examples is missing.
I wrote a tool called naml that is able to convert any Kubernetes YAML to raw Go. It is handy because it works using the version of Kubernetes you are running it against, and is compiled with the latest version of the Kubernetes code base.
If you wanted to create a Job, and see valid Go for the job it would look like this. Creating a Job beeps with container image boops
[nova#emma ~]$ kubectl create job beeps --image boops
job.batch/beeps created
[nova#emma ~]$
Naml will out a working program by design, but you will also get the output you are looking for.
[nova#emma naml]$ kubectl get job beeps -o yaml | naml codify
// Copyright ยฉ 2021 Kris Nรณva <kris#nivenly.com>
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
// โโโโ โโโ โโโโโโ โโโโ โโโโโโโ
// โโโโโ โโโโโโโโโโโโโโโโ โโโโโโโโ
// โโโโโโ โโโโโโโโโโโโโโโโโโโโโโโโโ
// โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
// โโโ โโโโโโโโโ โโโโโโ โโโ โโโโโโโโโโโ
// โโโ โโโโโโโโ โโโโโโ โโโโโโโโโโโ
//
package main
import (
"context"
"fmt"
"os"
"github.com/hexops/valast"
batchv1 "k8s.io/api/batch/v1"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/types"
"github.com/kris-nova/naml"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/client-go/kubernetes"
)
// Version is the current release of your application.
var Version string = "0.0.1"
func main() {
// Load the application into the NAML registery
// Note: naml.Register() can be used multiple times.
naml.Register(NewApp("App", "Application autogenerated from NAML v0.3.1"))
// Run the generic naml command line program with
// the application loaded.
err := naml.RunCommandLine()
if err != nil {
fmt.Println(err.Error())
os.Exit(1)
}
}
// App is a very important grown up business application.
type App struct {
metav1.ObjectMeta
description string
objects []runtime.Object
// ----------------------------------
// Add your configuration fields here
// ----------------------------------
}
// NewApp will create a new instance of App.
//
// See https://github.com/naml-examples for more examples.
//
// This is where you pass in fields to your application (similar to Values.yaml)
// Example: func NewApp(name string, example string, something int) *App
func NewApp(name, description string) *App {
return &App{
description: description,
ObjectMeta: metav1.ObjectMeta{
Name: name,
ResourceVersion: Version,
},
// ----------------------------------
// Add your configuration fields here
// ----------------------------------
}
}
func (a *App) Install(client *kubernetes.Clientset) error {
var err error
beepsJob := &batchv1.Job{
TypeMeta: metav1.TypeMeta{
Kind: "Job",
APIVersion: "batch/batchv1",
},
ObjectMeta: metav1.ObjectMeta{
Name: "beeps",
Namespace: "default",
UID: types.UID("650e4f36-3316-4506-bbe0-1e34c13742cf"),
ResourceVersion: "3231200",
Generation: 1,
Labels: map[string]string{
"controller-uid": "650e4f36-3316-4506-bbe0-1e34c13742cf",
"job-name": "beeps",
},
},
Spec: batchv1.JobSpec{
Parallelism: valast.Addr(int32(1)).(*int32),
Completions: valast.Addr(int32(1)).(*int32),
BackoffLimit: valast.Addr(int32(6)).(*int32),
Selector: &metav1.LabelSelector{MatchLabels: map[string]string{
"controller-uid": "650e4f36-3316-4506-bbe0-1e34c13742cf",
}},
Template: corev1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{
"controller-uid": "650e4f36-3316-4506-bbe0-1e34c13742cf",
"job-name": "beeps",
}},
Spec: corev1.PodSpec{
Containers: []corev1.Container{corev1.Container{
Name: "beeps",
Image: "boops",
TerminationMessagePath: "/dev/termination-log",
TerminationMessagePolicy: corev1.TerminationMessagePolicy("File"),
ImagePullPolicy: corev1.PullPolicy("Always"),
}},
RestartPolicy: corev1.RestartPolicy("Never"),
TerminationGracePeriodSeconds: valast.Addr(int64(30)).(*int64),
DNSPolicy: corev1.DNSPolicy("ClusterFirst"),
SecurityContext: &corev1.PodSecurityContext{},
SchedulerName: "default-scheduler",
},
},
CompletionMode: valast.Addr(batchv1.CompletionMode("NonIndexed")).(*batchv1.CompletionMode),
Suspend: valast.Addr(false).(*bool),
},
}
a.objects = append(a.objects, beepsJob)
if client != nil {
_, err = client.BatchV1().Jobs("default").Create(context.TODO(), beepsJob, metav1.CreateOptions{})
if err != nil {
return err
}
}
return err
}
func (a *App) Uninstall(client *kubernetes.Clientset) error {
var err error
if client != nil {
err = client.BatchV1().Jobs("default").Delete(context.TODO(), "beeps", metav1.DeleteOptions{})
if err != nil {
return err
}
}
return err
}
func (a *App) Description() string {
return a.description
}
func (a *App) Meta() *metav1.ObjectMeta {
return &a.ObjectMeta
}
func (a *App) Objects() []runtime.Object {
return a.objects
}
While the kubernetes golang api example for out-of-cluster authentication works fine, and creating a service account and exporting the bearer token works great, it feels silly to write the pieces to a temporary file only to tell the API to read it. Is there an API way to pass these pieces as an object rather than write to a file?
clusterData := map[string]string{
"BEARER_TOKEN": bearerToken,
"CA_DATA": clusterCA,
"ENDPOINT": clusterUrl,
}
const kubeConfigTmpl = `
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: {{.CA_DATA}}
server: {{.HOST_IP_ADDRESS}}
name: kubernetes
contexts:
- context:
cluster: kubernetes
namespace: default
user: lamdba-serviceaccount-default-kubernetes
name: lamdba-serviceaccount-default-kubernetes
current-context: lamdba-serviceaccount-default-kubernetes
kind: Config
preferences: {}
users:
- name: lamdba-serviceaccount-default-kubernetes
user:
token: {{.BEARER_TOKEN}}
`
t := template.Must(template.New("registration").Parse(kubeConfigTmpl))
buf := &bytes.Buffer{}
if err := t.Execute(buf, clusterData); err != nil {
panic(err)
}
registrationPayload := buf.String()
d1 := []byte(registrationPayload)
err := ioutil.WriteFile("/tmp/config", d1, 0644)
The rest.Config struct passed to the NewFromConfig client constructors lets you specify bearer tokens and/or client certificate/key data directly.
Looking at the source code, this should work:
// error handling omitted for brevity
cc, _ := clientcmd.NewClientConfigFromBytes([]byte(d1))
config, _ := cc.ClientConfig()
clientset, _ := kubernetes.NewForConfig(config)
How can I deserialize a Kubernetes YAML file into an Go struct? I took a look into the kubectl code, but somehow I get an error for every YAML file:
no kind "Deployment" is registered for version "apps/v1beta1"
This is an MWE:
package main
import (
"fmt"
"k8s.io/client-go/pkg/api"
)
var service = `
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: my-nginx
spec:
replicas: 2
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- name: my-nginx
image: nginx
ports:
- containerPort: 80
`
func main() {
decode := api.Codecs.UniversalDecoder().Decode
//decode := api.Codecs.UniversalDeserializer().Decode
obj, _, err := decode([]byte(service), nil, nil)
if err != nil {
panic(err)
}
fmt.Printf("%#v\n", obj)
}
I am using client version 2.0.0. The glide.yaml looks like this:
package: test/stackoverflow
import:
- package: k8s.io/client-go
version: ^2.0.0
These are the references to kubectl:
https://github.com/kubernetes/kubernetes/blob/43ac38e29e6ecf83e78bc7c5d9f804310b051c95/pkg/kubectl/cmd/apply.go#L637
https://github.com/kubernetes/kubernetes/blob/43ac38e29e6ecf83e78bc7c5d9f804310b051c95/pkg/kubectl/cmd/util/factory_client_access.go#L205-L213
Unfortunately, the docs are very confusing to me, so I have no idea how to tackle this problem.
Edit:
This problem also exists with other resource types:
no kind "Service" is registered for version "v1"
You need to import _ "k8s.io/client-go/pkg/apis/extensions/install" otherwise the schema is empty, see also docs.
The complete working example is:
$ go get -u github.com/golang/dep/cmd/dep
$ dep init
$ go run main.go
With the following main.go:
package main
import (
"fmt"
"k8s.io/client-go/pkg/api"
_ "k8s.io/client-go/pkg/api/install"
_ "k8s.io/client-go/pkg/apis/extensions/install"
)
var deployment = `
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: my-nginx
spec:
replicas: 2
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- name: my-nginx
image: nginx
ports:
- containerPort: 80
`
func main() {
// decode := api.Codecs.UniversalDecoder().Decode
decode := api.Codecs.UniversalDeserializer().Decode
obj, _, err := decode([]byte(deployment), nil, nil)
if err != nil {
fmt.Printf("%#v", err)
}
fmt.Printf("%#v\n", obj)
}
Note that I also imported _ "k8s.io/client-go/pkg/api/install" for you so that you can use objects in v1 such as pods or services.
EDIT: Kudos to my colleague Stefan Schimanski who proposed the initial solution.
I've been using api machinery'sk8s.io/apimachinery/pkg/util/yaml to decode kubernete's deployment and service manifests.
import (
"fmt"
"bytes"
appsv1 "k8s.io/api/apps/v1"
k8Yaml "k8s.io/apimachinery/pkg/util/yaml"
)
...
d := &appsv1.Deployment{}
dec := k8Yaml.NewYAMLOrJSONDecoder(bytes.NewReader([]byte(deploymentManifest)), 1000)
if err := dec.Decode(&d); err != nil {
return nil, err
}
fmt.Printf("%+v", d)
import (
"fmt"
"gopkg.in/yaml.v2"
"log"
//corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/runtime/serializer/json"
"k8s.io/client-go/kubernetes/scheme"
v1alpha1 "github.com/argoproj/argo-cd/pkg/apis/application/v1alpha1"
)
....
func ParseYaml2(yaml []byte) (v1alpha1.Application, error) {
// Create a YAML serializer. JSON is a subset of YAML, so is supported too.
s := json.NewYAMLSerializer(json.DefaultMetaFactory, scheme.Scheme,
scheme.Scheme)
// Decode the YAML to an object.
var app v1alpha1.Application
_, _, err := s.Decode(yaml, nil, &app)
if err != nil {
panic(err)
}
//fmt.Printf("%#v\n", app)
return app, err
}
---
go.mod
// https://github.com/argoproj/argo-cd/issues/4055
replace github.com/argoproj/argo-cd => github.com/argoproj/argo-cd v1.5.5
var yaml2 = []byte(`
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
...
var app v1alpha1.Application
app,err := ParseYaml2(yaml2)
// Types from https://github.com/argoproj/argo-cd/blob/master/pkg/apis/application/v1alpha1/types.go
fmt.Printf("--- t:\n%s\n\n", app.Spec.Source.Path)
fmt.Printf("--- t:\n%s\n\n", app.Spec.Source.Helm.ValueFiles)
----