How to use dapr nats io jetstream with Go - go

I am using dapr ,to use its pub sub configuration and using dapr go sdk to publish the messages .
With redis its working fine ,but when I am using nats.io jetstream its giving me error stream doesn't exists which ,even I am creating stream too
pubsub.yml I am using to run the dapr side car
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: order_pub_sub
namespace: default
spec:
type: pubsub.jetstream
version: v1
metadata:
- name: natsURL
value: "nats://localhost:4222"
- name: name
value: "connection name"
- name: durableName
value: "consumer durable name"
- name: queueGroupName
value: "queue group name"
- name: startSequence
value: 1
- name: startTime # in Unix format
value: 1630349391
- name: deliverAll
value: false
- name: flowControl
value: false
And this is the go code
//dependencies
import (
"context"
"log"
"math/rand"
"time"
"strconv"
dapr "github.com/dapr/go-sdk/client"
)
//code
var (
PUBSUB_NAME = "order_pub_sub"
TOPIC_NAME = "orders"
)
func main() {
for i := 0; i < 10; i++ {
time.Sleep(5000)
orderId := rand.Intn(1000-1) + 1
client, err := dapr.NewClient()
if err != nil {
panic(err)
}
defer client.Close()
ctx := context.Background()
//Using Dapr SDK to publish a topic
if err := client.PublishEvent(ctx, PUBSUB_NAME, TOPIC_NAME, []byte(strconv.Itoa(orderId)));
err != nil {
panic(err)
}
log.Println("Published data: " + strconv.Itoa(orderId))
}
}
I just want to know what will be the relevant names in nats ,like for topic and pubsub_name ,Or anyone able can help ?

Related

Error while creating a pod using kubernetes client API

I'm trying to create a pod using Kubernetes client api in Go and I've been getting below error in TravisCI,
ERRO Running error: buildir: analysis skipped: errors in package: [/home/travis/gopath/src/github.com/pravarag/test-repo/check_pod.go:25:70: cannot use desiredPod (variable of type *"k8s.io/api/core/v1".Pod) as context.Context value in argument to s.kubeClient.CoreV1().Pods(desiredPod.Namespace).Create: missing method Deadline /home/travis/gopath/src/github.com/pravarag/test-repo/check_pod.go:25:80: too few arguments in call to s.kubeClient.CoreV1().Pods(desiredPod.Namespace).Create
Below is the code,
import (
"fmt"
"go.uber.org/zap"
core "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
func (s *server) createPod() {
// build the pod definition
desiredPod := getPodObjet()
pod, err := s.kubeClient.CoreV1().Pods(desiredPod.Namespace).Create(desiredPod)
if err != nil {
s.log.Fatal("Failed to create the static pod", zap.Error(err))
}
fmt.Println("Created Pod: ", pod.Name)
}
func getPodObjet() *core.Pod {
pod := &core.Pod{
ObjectMeta: metav1.ObjectMeta{
Name: "test-pod",
Namespace: "default",
Labels: map[string]string{
"app": "test-pod",
},
},
Spec: core.PodSpec{
Containers: []core.Container{
{
Name: "busybox",
Image: "busybox",
ImagePullPolicy: core.PullIfNotPresent,
Command: []string{
"sleep",
"3600",
},
},
},
},
}
return pod
}
I tried to check what that error is pointing to and it seems, the actual pod Interface in K8s client code here: https://godoc.org/k8s.io/client-go/kubernetes/typed/core/v1#PodInterface
is expecting 3 arguments: one is "context.Context", "pod *v1.Pod" and "opts metav1.CreateOptions"
I tried to pass the values as:
pod, err :=s.kubeClient.CoreV1().Pods(desiredPod.Namespace).Create(context.Context, desiredPod, opts metav1.CreateOptions{})
But that doesn't work also. Even in the IDE, the code lint is pointing to missing arguments but I've seen couple of examples used to create a pod in above mentioned way that worked previously.
just use context.TODO() as argument to pass context.
try this one.
pod, err := s.kubeClient.CoreV1().Pods(desiredPod.Namespace).Create( context.TODO(), desiredPod , metav1.CreateOptions{})
here is the updated code:
import (
"fmt"
"context"
"go.uber.org/zap"
core "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
func (s *server) createPod() {
// build the pod definition
desiredPod := getPodObjet()
pod, err := s.kubeClient.CoreV1().Pods(desiredPod.Namespace).Create( context.TODO(), desiredPod , metav1.CreateOptions{})
if err != nil {
s.log.Fatal("Failed to create the static pod", zap.Error(err))
}
fmt.Println("Created Pod: ", pod.Name)
}
func getPodObjet() *core.Pod {
pod := &core.Pod{
ObjectMeta: metav1.ObjectMeta{
Name: "test-pod",
Namespace: "default",
Labels: map[string]string{
"app": "test-pod",
},
},
Spec: core.PodSpec{
Containers: []core.Container{
{
Name: "busybox",
Image: "busybox",
ImagePullPolicy: core.PullIfNotPresent,
Command: []string{
"sleep",
"3600",
},
},
},
},
}
return pod
}

Generic client.Get for custom Kubernetes GO operator

In a custom Kubernetes operator implemented with the operator-sdk in golang is it possible to call the custom API directly and retrieve the object as YAML?
For example. I have a custom resource
apiVersion: test.com/v1alpha1
kind: TEST
metadata::
name: example-test
spec:
replicas: 3
randomname: value
I don't know ahead of time what the fields in the spec are going to be apart from replicas. So I am not able to create a go type that includes structs to hold the entries.
So rather than doing:
instance := &testv1alpha1.Test{}
err := r.client.Get(context.TODO(), nameSpaceName, instance)
I want to be able to do something like:
instanceYAML := genericContainer{}
err := r.client.GetGeneric(context.TODO(), nameSpaceName, instance)
and then parse the instanceYAML to check the entries.
This is called the "unstructured" client. The docs are pretty light so I recommend looking over the tests as examples https://github.com/kubernetes-sigs/controller-runtime/blob/ea32729106c995d9df310ac4731c2061490addfb/pkg/client/client_test.go#L1536-L1566
Using the unstructured type client all the operation done on the CoreAPI resource can be done on the CustomResourceDefinition(CRD)
package util
import (
"context"
"encoding/json"
"strconv"
"github.com/go-logr/logr"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
"k8s.io/apimachinery/pkg/runtime/schema"
types "k8s.io/apimachinery/pkg/types"
_ "k8s.io/client-go/plugin/pkg/client/auth"
"sigs.k8s.io/controller-runtime/pkg/client"
)
var reqLogger logr.Logger
func CRUDonCRD(c client.Client) {
u := createUnstructuredObject(c)
GetCR(u, c)
PatchCR(u, c, 1)
DeleteCR(u, c)
}
func createUnstructuredObject(c client.Client) *unstructured.Unstructured {
u := &unstructured.Unstructured{}
u.SetGroupVersionKind(schema.GroupVersionKind{
// Group: "<crd group name>",
// Kind: "<crd kind>",
// Version: "<crd version>",
Group: "com.cumulus.netq.operator.transport",
Kind: "TransportOperator",
Version: "v1",
})
_ = c.Get(context.Background(), client.ObjectKey{
// Namespace: "<namespace>",
// Name: "cr name",
Namespace: "default",
Name: "transport-operator",
}, u)
return u
}
type patchStringValue struct {
Op string `json:"op"`
Path string `json:"path"`
Value string `json:"value"`
}
func PatchCR(u *unstructured.Unstructured, c client.Client, replicaCount int) error {
payload := []patchStringValue{{
Op: "replace",
Path: "/spec/replicas/replicasOpta",
Value: strconv.Itoa(replicaCount),
}}
payloadBytes, _ := json.Marshal(payload)
err := c.Patch(context.Background(), u, client.RawPatch(types.JSONPatchType, payloadBytes))
if err != nil {
reqLogger.Error(err, "error occured while patching")
}
reqLogger.Info("Patching is successful", "Patched Transport Operator Object", u)
return err
}
func GetCR(u *unstructured.Unstructured, c client.Client) {
// key := client.ObjectKey{Namespace: "default", Name: "<cr-name>"}
key := client.ObjectKey{Namespace: "default", Name: "transport-operator"}
err := c.Get(context.Background(), key, u)
if err != nil {
reqLogger.Error(err, "error occured while getting the resource")
}
reqLogger.Info("Got the resource", "Resource Object", u)
}
func DeleteCR(u *unstructured.Unstructured, c client.Client) {
err := c.Delete(context.Background(), u)
if err != nil {
reqLogger.Error(err, "error occured while deleting the resource")
}
reqLogger.Info("Resource deleted", "Resource ", u)
}
Here is what I did in the Operator SDK to get a object from kubernetes.
Ref: https://github.com/kubernetes-sigs/controller-runtime/blob/master/pkg/client/example_test.go#L57
func (r *ConfigDeploymentReconciler) findObjectsForConfigMap(configMap client.Object) []reconcile.Request {
pod := &corev1.Pod{}
_ = r.Client.Get(context.Background(), client.ObjectKey{
Namespace: "prometheus",
Name: "prometheus-node-exporter-7vxqs",
}, pod)
fmt.Println("Got this pod", pod.String())
return nil
}

How do I get/update a Kubernetes custom resource from a go program?

I'm looking for the go equivalent of:
kubectl get some-custom-resource-kind some-custom-resource -o yaml > file.yaml
Modify the yaml file...
kubectl apply -f file.yaml
Kubernetes has a client go library for standard resource kinds.
And various vendors have client go libraries for their custom resources.
But I need to get/update a resource kind that doesn't have a publicly available client go library. The logic is implemented in bash script today and I'm trying to move that function to a go controller.
Seems like it there should be a straightforward way in go to do the equivalent of kubectl.
Thanks, Paul
For any type, including your CRDs, use client.Client.
From the documentation:
// Using a typed object.
pod := &corev1.Pod{}
// c is a created client.
_ = c.Get(context.Background(), client.ObjectKey{
Namespace: "namespace",
Name: "name",
}, pod)
pod.SetFinalizers(append(pod.GetFinalizers(), "new-finalizer"))
_ = c.Update(context.Background(), pod)
// Using a unstructured object.
u := &unstructured.Unstructured{}
u.SetGroupVersionKind(schema.GroupVersionKind{
Group: "apps",
Kind: "Deployment",
Version: "v1",
})
_ = c.Get(context.Background(), client.ObjectKey{
Namespace: "namespace",
Name: "name",
}, u)
u.SetFinalizers(append(u.GetFinalizers(), "new-finalizer"))
_ = c.Update(context.Background(), u)
You could just as easily swap in SomeCustomResourceKind:
myCR := &v1alpha1.SomeCustomResourceKind{}
// c is a created client.Client
_ = c.Get(context.TODO(), client.ObjectKey{
Namespace: "namespace",
Name: "some-custom-resource", }, myCR)
myCR.MyProperty = "NewValue"
_ = c.Update(context.TODO(), myCR)
You mentioned you're trying to move this functionality from a bash script to a Go controller, so it would be worth checking out the Kubebuilder project, which can scaffold out a controller for you (and any additional APIs you might need). It creates fully functional controllers with the controller-runtime Client and wires up all the reconciliation logic to manage your CRDs.
I am creating a CRD (securitycontextconstraint) as follows:
import(
v1 "github.com/openshift/api/security/v1"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/resource"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
func (c *ClusterClient) CreateSCC() {
name := "ssc-test"
scc := v1.SecurityContextConstraints{
TypeMeta: metav1.TypeMeta{
Kind: "SecurityContextConstraints",
APIVersion: "security.openshift.io/v1",
},
ObjectMeta: metav1.ObjectMeta{
Name: name,
},
RequiredDropCapabilities: []corev1.Capability{"KILL", "MKNOD", "SETUID", "SETGID"},
AllowHostPorts: false,
AllowPrivilegedContainer: false,
AllowHostDirVolumePlugin: false,
AllowHostIPC: false,
ReadOnlyRootFilesystem: false,
DefaultAddCapabilities: nil,
AllowHostPID: false,
AllowHostNetwork: false,
AllowedCapabilities: nil,
Volumes: []v1.FSType{v1.FSTypeConfigMap, v1.FSTypeDownwardAPI, v1.FSTypeEmptyDir, v1.FSTypePersistentVolumeClaim, v1.FSProjected, v1.FSTypeSecret},
SELinuxContext: v1.SELinuxContextStrategyOptions{
Type: v1.SELinuxStrategyMustRunAs,
},
RunAsUser: v1.RunAsUserStrategyOptions{
Type: v1.RunAsUserStrategyMustRunAsRange,
},
SupplementalGroups: v1.SupplementalGroupsStrategyOptions{
Type: v1.SupplementalGroupsStrategyRunAsAny,
},
Users: []string{},
Groups: []string{"system:authenticated"},
}
// To check if scc is already created or not
_, err := c.kubeClient.CoreV1().RESTClient().
Get().
AbsPath("/apis/security.openshift.io/v1/securitycontextconstraints/ssc-test").
DoRaw(context.Background())
// scc is not present in cluster
if err != nil {
//Creating SCC
_, err = c.kubeClient.CoreV1().RESTClient().
Post().
AbsPath("/apis/security.openshift.io/v1").
Resource("securitycontextconstraints").
// VersionedParams(&metav1.CreateOptions{}, scheme.ParameterCodec).
Body(&scc).
DoRaw(context.Background())
if err != nil {
fmt.Printf("Failed to create SecurityContextConstraints: %v\n", err)
// return Failure, err
}
fmt.Printf("Successfully created SecurityContextConstraints %s\n", name)
}
}

Match data from structs objects

in the following code (which I re-used from previous question) we need to match objects according to matching keys values
The keys are the value of
From model internalConfig
type
From external config
app-type
In case we have 'match' (in the value) we need to replace the data from the external config and update the internal config
e.g. if for zone2 app-type=app1 is equal to internal config type=app1 we need to change the content from external config to be
from
- name: TEST
type: app1
target: ./
to (since app1 is equal in internal and external config- we are updating the data from external to internal)
- name: TEST
type: AZURE
target: ./
given the above data the internal config should look after the match like
At the end it should look like
- name: TEST
type: AZURE
target: ./
- name: TEST3
type: HEROKU
target: ./
- name: TEST2
type: app3
target: ./
- name: TEST2
type: AZURE
target: ./
Im trying the following but I wasn’t able to match all the objects in case I’ve more then one entry to the same key
e.g. app1 “key” is in the first and the last entry, so my code is update only the first entry
- name: TEST
type: app1
target: ./
- name: TEST3
type: app2
target: ./
- name: TEST2
type: app3
target: ./
- name: TEST2
type: app1
target: ./
Any idea how could I resolve this ? this is what i've tried which works only on the first match
package main
import (
"fmt"
"gopkg.in/yaml.v2"
"log"
)
type InternalConfig struct {
Models []Imodel `yaml:"models"`
}
type Imodel struct {
Name string `yaml:"name"`
Type string `yaml:"type"`
Target string `yaml:"target"`
}
var internalConfig = []byte(`
models:
- name: TEST
type: app1
target: ./
- name: TEST3
type: app2
target: ./
- name: TEST2
type: app3
target: ./
- name: TEST2
type: app1
target: ./
`)
type ExternalConfig struct {
Landscape Zone `yaml:"Landscape"`
}
type Zone struct {
Zone string `yaml:"zone"`
Models []Emodel `yaml:"models"`
}
type Emodel struct {
AppType string `yaml:"app-type"`
ServiceType string `yaml:"service-type"`
}
var externalConfig = []byte(`
Landscape:
zone: zone1
models:
- app-type: app1
service-type: GCP
- app-type: app2
service-type: AMAZON
zone: zone2
models:
- app-type: app1
service-type: AZURE
- app-type: app2
service-type: HEROKU
`)
func main() {
iCfgYaml := InternalConfig{}
if err := yaml.Unmarshal(internalConfig, &iCfgYaml); err != nil {
log.Fatalf("error in model InternalConfig: %v", err)
}
eCfgYaml := ExternalConfig{}
if err := yaml.Unmarshal(externalConfig, &eCfgYaml); err != nil {
log.Fatalf("error in model ExternalConfig: %v", err)
}
if err := UpdateInternalState(&iCfgYaml, eCfgYaml, "zone2"); err != nil {
log.Fatalf("Error updating internal config: %v", err)
}
fmt.Printf("DBG: %+v\n", iCfgYaml)
}
func UpdateInternalState(iCfg *InternalConfig, eCfg ExternalConfig, zone string) error {
ilm := make(map[string]int)
for i, v := range iCfg.Models {
ilm[v.Type] = i
}
for _, em := range eCfg.Landscape.Models {
if eCfg.Landscape.Zone == zone {
if i, ok := ilm[em.AppType]; ok {
iCfg.Models[i].Type = em.ServiceType
}
}
}
return nil
}
if you run the code above, this is the data you will get
{Models:[
{Name:TEST Type:app1 Target:./}
{Name:TEST3 Type:HEROKU Target:./}
{Name:TEST2 Type:app3 Target:./}
{Name:TEST2 Type:AZURE Target:./}]}
The first entry doesn't updated
https://play.golang.org/p/5f3Ad9Z7fqY
If needs more clarification please let me know

How to deserialize Kubernetes YAML file

How can I deserialize a Kubernetes YAML file into an Go struct? I took a look into the kubectl code, but somehow I get an error for every YAML file:
no kind "Deployment" is registered for version "apps/v1beta1"
This is an MWE:
package main
import (
"fmt"
"k8s.io/client-go/pkg/api"
)
var service = `
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: my-nginx
spec:
replicas: 2
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- name: my-nginx
image: nginx
ports:
- containerPort: 80
`
func main() {
decode := api.Codecs.UniversalDecoder().Decode
//decode := api.Codecs.UniversalDeserializer().Decode
obj, _, err := decode([]byte(service), nil, nil)
if err != nil {
panic(err)
}
fmt.Printf("%#v\n", obj)
}
I am using client version 2.0.0. The glide.yaml looks like this:
package: test/stackoverflow
import:
- package: k8s.io/client-go
version: ^2.0.0
These are the references to kubectl:
https://github.com/kubernetes/kubernetes/blob/43ac38e29e6ecf83e78bc7c5d9f804310b051c95/pkg/kubectl/cmd/apply.go#L637
https://github.com/kubernetes/kubernetes/blob/43ac38e29e6ecf83e78bc7c5d9f804310b051c95/pkg/kubectl/cmd/util/factory_client_access.go#L205-L213
Unfortunately, the docs are very confusing to me, so I have no idea how to tackle this problem.
Edit:
This problem also exists with other resource types:
no kind "Service" is registered for version "v1"
You need to import _ "k8s.io/client-go/pkg/apis/extensions/install" otherwise the schema is empty, see also docs.
The complete working example is:
$ go get -u github.com/golang/dep/cmd/dep
$ dep init
$ go run main.go
With the following main.go:
package main
import (
"fmt"
"k8s.io/client-go/pkg/api"
_ "k8s.io/client-go/pkg/api/install"
_ "k8s.io/client-go/pkg/apis/extensions/install"
)
var deployment = `
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: my-nginx
spec:
replicas: 2
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- name: my-nginx
image: nginx
ports:
- containerPort: 80
`
func main() {
// decode := api.Codecs.UniversalDecoder().Decode
decode := api.Codecs.UniversalDeserializer().Decode
obj, _, err := decode([]byte(deployment), nil, nil)
if err != nil {
fmt.Printf("%#v", err)
}
fmt.Printf("%#v\n", obj)
}
Note that I also imported _ "k8s.io/client-go/pkg/api/install" for you so that you can use objects in v1 such as pods or services.
EDIT: Kudos to my colleague Stefan Schimanski who proposed the initial solution.
I've been using api machinery'sk8s.io/apimachinery/pkg/util/yaml to decode kubernete's deployment and service manifests.
import (
"fmt"
"bytes"
appsv1 "k8s.io/api/apps/v1"
k8Yaml "k8s.io/apimachinery/pkg/util/yaml"
)
...
d := &appsv1.Deployment{}
dec := k8Yaml.NewYAMLOrJSONDecoder(bytes.NewReader([]byte(deploymentManifest)), 1000)
if err := dec.Decode(&d); err != nil {
return nil, err
}
fmt.Printf("%+v", d)
import (
"fmt"
"gopkg.in/yaml.v2"
"log"
//corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/runtime/serializer/json"
"k8s.io/client-go/kubernetes/scheme"
v1alpha1 "github.com/argoproj/argo-cd/pkg/apis/application/v1alpha1"
)
....
func ParseYaml2(yaml []byte) (v1alpha1.Application, error) {
// Create a YAML serializer. JSON is a subset of YAML, so is supported too.
s := json.NewYAMLSerializer(json.DefaultMetaFactory, scheme.Scheme,
scheme.Scheme)
// Decode the YAML to an object.
var app v1alpha1.Application
_, _, err := s.Decode(yaml, nil, &app)
if err != nil {
panic(err)
}
//fmt.Printf("%#v\n", app)
return app, err
}
---
go.mod
// https://github.com/argoproj/argo-cd/issues/4055
replace github.com/argoproj/argo-cd => github.com/argoproj/argo-cd v1.5.5
var yaml2 = []byte(`
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
...
var app v1alpha1.Application
app,err := ParseYaml2(yaml2)
// Types from https://github.com/argoproj/argo-cd/blob/master/pkg/apis/application/v1alpha1/types.go
fmt.Printf("--- t:\n%s\n\n", app.Spec.Source.Path)
fmt.Printf("--- t:\n%s\n\n", app.Spec.Source.Helm.ValueFiles)
----

Resources