when I was using go111, I had traces of all my Datastore calls (similar to image below). But as soon as I upgraded to go115 and started using cloud.google.com/go/datastore, I lost this information completely. I tried to set up telemetry by adding in my main:
projectID := os.Getenv("GOOGLE_CLOUD_PROJECT")
exporter, err := texporter.NewExporter(texporter.WithProjectID(projectID))
if err != nil {
log.Fatalf(bgCtx, "texporter.NewExporter of '%v': %v", projectID, err)
}
tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(exporter))
defer tp.ForceFlush(bgCtx)
otel.SetTracerProvider(tp)
But this didn't work. Am I missing anything to tell the datastore library to export those calls?
Thank you!
I finally found https://github.com/GoogleCloudPlatform/golang-samples/blob/master/trace/trace_quickstart/main.go
and realized I was missing the following:
trace.RegisterExporter(exporter)
This solved my problem. Then I also added the following on localhost
trace.ApplyConfig(trace.Config{DefaultSampler: trace.AlwaysSample()})
To make sure all requests are traced:
httpHandler := &ochttp.Handler{
// Use the Google Cloud propagation format.
Propagation: &propagation.HTTPFormat{},
}
if err := http.ListenAndServe(":"+port, httpHandler); err != nil {
Related
I'm trying to get kafka producer/consumer metrics using sarama. But I'm unable to find any example on how to do the same. Can someone provide a sample implementation example?
I'm using the following code to get metrics of a broker. But what should be the config to get metrics of a producer/consumer. I'm assuming that it will not be the same. Correct me if I'm wrong
saramaConfig := sarama.NewConfig()
saramaConfig.Version = <BrokerVersion>
client, err := sarama.NewClient(<brokerAddresses>, saramaConfig)
if err != nil {
log.Println("Unable to create sarama client")
panic(err)
}
config := client.Config()
MeanIncomingByteRate := metrics.GetOrRegisterMeter(getMetricNameForTopic("incoming-byte-rate", topic), config.MetricRegistry).RateMean(),
I've never worked with. But I will try to answer.
You should register the go-metrics registry to the config.
This is an example of how to create a registry and use it (https://pkg.go.dev/gopkg.in/Shopify/sarama.v2#example-Config-Metrics):
// Our application registry
appMetricRegistry := metrics.NewRegistry()
appGauge := metrics.GetOrRegisterGauge("m1", appMetricRegistry)
appGauge.Update(1)
config := NewConfig()
// Use a prefix registry instead of the default local one
config.MetricRegistry = metrics.NewPrefixedChildRegistry(appMetricRegistry, "sarama.")
// Simulate a metric created by sarama without starting a broker
saramaGauge := metrics.GetOrRegisterGauge("m2", config.MetricRegistry)
saramaGauge.Update(2)
metrics.WriteOnce(appMetricRegistry, os.Stdout)
So you should create the registry and set it to the config
appMetricRegistry := metrics.NewRegistry()
appGauge := metrics.GetOrRegisterGauge("m1", appMetricRegistry)
appGauge.Update(1)
saramaConfig := sarama.NewConfig()
saramaConfig.Version = <BrokerVersion>
// set it here
saramaConfig.MetricRegistry = metrics.NewPrefixedChildRegistry(appMetricRegistry, "sarama.")
client, err := sarama.NewClient(<brokerAddresses>, saramaConfig)
if err != nil {
log.Println("Unable to create sarama client")
panic(err)
}
config := client.Config()
MeanIncomingByteRate := metrics.GetOrRegisterMeter(getMetricNameForTopic("incoming-byte-rate", topic), config.MetricRegistry).RateMean(),
I've recently shifted from python to golang. I had been using python to work with GCP.
I used to pass in the scopes and mention the discovery client I wanted to create like this :
def get_client(scopes, api, version="v1"):
service_account_json = os.environ.get("SERVICE_ACCOUNT_KEY_JSON", None)
if service_account_json is None:
sys.exit("Exiting !!! No SSH_KEY_SERVICE_ACCOUNT env var found.")
credentials = service_account.Credentials.from_service_account_info(
json.loads(b64decode(service_account_json)), scopes=scopes
)
return discovery.build(api, version, credentials=credentials, cache_discovery=False)
And this would create my desired discovery client, whether it be compute engine service or sqladmin
However in go I don't seem to find this.
I found this : https://pkg.go.dev/google.golang.org/api/discovery/v1
For any client that I want to create I would've to import that and then create that, like this :
https://cloud.google.com/resource-manager/reference/rest/v1/projects/list#examples
package main
import (
"fmt"
"log"
"golang.org/x/net/context"
"golang.org/x/oauth2/google"
"google.golang.org/api/cloudresourcemanager/v1"
)
func main() {
ctx := context.Background()
c, err := google.DefaultClient(ctx, cloudresourcemanager.CloudPlatformScope)
if err != nil {
log.Fatal(err)
}
cloudresourcemanagerService, err := cloudresourcemanager.New(c)
if err != nil {
log.Fatal(err)
}
req := cloudresourcemanagerService.Projects.List()
if err := req.Pages(ctx, func(page *cloudresourcemanager.ListProjectsResponse) error {
for _, project := range page.Projects {
// TODO: Change code below to process each `project` resource:
fmt.Printf("%#v\n", project)
}
return nil
}); err != nil {
log.Fatal(err)
}
}
So I've to import each client library to get the client for that.
"google.golang.org/api/cloudresourcemanager/v1"
There's no dynamic creation of it.
Is it even possible, cause go is strict type checking 🤔
Thanks.
No, this is not possible with the Golang Google Cloud library.
You've nailed the point on the strict type checking, as it would definitely defeat the benefits of compile time type checking. It would also be a bad Golang practice to return different objects with different signatures, as we don't do duck typing and instead we rely on interface contracts.
Golang is boring and verbose, and it's like that by design :)
The kubernetes go client has tons of methods and I can't find how I can get the current CPU & RAM usage of a specific (or all pods).
Can someone tell me what methods I need to call to get the current usage for pods & nodes?
My NodeList:
nodes, err := clientset.CoreV1().Nodes().List(metav1.ListOptions{})
Kubernetes Go Client: https://github.com/kubernetes/client-go
Metrics package: https://github.com/kubernetes/kubernetes/tree/master/staging/src/k8s.io/metrics
As far as I got the metrics server implements the Kubernetes metrics package in order to fetch the resource usage from pods and nodes, but I couldn't figure out where & how they do it: https://github.com/kubernetes-incubator/metrics-server
It is correct that go-client does not have support for metrics type, but in the metrics package there is a pregenerated client that can be used for fetching metrics objects and assign them right away to the appropriate structure. The only thing you need to do first is to generate a config and pass it to metrics client. So a simple client for metrics would look like this:
package main
import (
"k8s.io/client-go/tools/clientcmd"
metrics "k8s.io/metrics/pkg/client/clientset/versioned"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
func main() {
var kubeconfig, master string //empty, assuming inClusterConfig
config, err := clientcmd.BuildConfigFromFlags(master, kubeconfig)
if err != nil{
panic(err)
}
mc, err := metrics.NewForConfig(config)
if err != nil {
panic(err)
}
mc.MetricsV1beta1().NodeMetricses().Get("your node name", metav1.GetOptions{})
mc.MetricsV1beta1().NodeMetricses().List(metav1.ListOptions{})
mc.MetricsV1beta1().PodMetricses(metav1.NamespaceAll).List(metav1.ListOptions{})
mc.MetricsV1beta1().PodMetricses(metav1.NamespaceAll).Get("your pod name", metav1.GetOptions{})
}
Each of the above methods from metric client returns an appropriate structure (you can check those here) and an error (if any) which you should process according to your requirements.
here is an example.
package main
import (
"fmt"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/tools/clientcmd"
metrics "k8s.io/metrics/pkg/client/clientset/versioned"
)
func main() {
var kubeconfig, master string //empty, assuming inClusterConfig
config, err := clientcmd.BuildConfigFromFlags(master, kubeconfig)
if err != nil {
panic(err)
}
mc, err := metrics.NewForConfig(config)
if err != nil {
panic(err)
}
podMetrics, err := mc.MetricsV1beta1().PodMetricses(metav1.NamespaceAll).List(metav1.ListOptions{})
if err != nil {
fmt.Println("Error:", err)
return
}
for _, podMetric := range podMetrics.Items {
podContainers := podMetric.Containers
for _, container := range podContainers {
cpuQuantity, ok := container.Usage.Cpu().AsInt64()
memQuantity, ok := container.Usage.Memory().AsInt64()
if !ok {
return
}
msg := fmt.Sprintf("Container Name: %s \n CPU usage: %d \n Memory usage: %d", container.Name, cpuQuantity, memQuantity)
fmt.Println(msg)
}
}
}
The API you're looking for in new versions of Kubernetes (tested on mine as of 1.10.7) is the metrics.k8s.io/v1beta1 API route.
You can see it locally if you run a kubectl proxy and check http://localhost:8001/apis/metrics.k8s.io/v1beta1/pods and /nodes on your localhost.
I see where your confusion is though. At the time of writing, it does not look like the metrics/v1beta1 has a generated typed package (https://godoc.org/k8s.io/client-go/kubernetes/typed), and doesn't appear in the kubernetes.ClientSet object.
You can hit all available endpoints directly though the rest.RestClient object, and just specify metrics/v1beta1 as the versionedAPIPath, which will be more work and less convenient than the nicely wrapped ClientSet, but I'm not sure how long it'll take before that API shows up in that interface.
I am new to fireStore and trying to retrieve document values but it isn't working as I would expect. I am getting a "permission denied" when trying to access through my IDE and golang.
Code:
func TestConnectToCollection(t *testing.T) {
ctx := context.Background()
client, err := firestore.NewClient(ctx, "<my-Project-ID>")
if err != nil {
// TODO: Handle error.
}
defer client.Close()
doc := client.Doc("profile/test3")
fmt.Println(doc)
fmt.Println(doc.ID)
iter := client.Collection("profile").Documents(ctx)
for {
doc, error := iter.Next()
if error == iterator.Done {
break
}
if error != nil {
fmt.Println(error.Error())
} else {
fmt.Println(doc.Data())
}
}
}
Output:
&{0xc0001725a0 projects/<project-id>/databases/(default)/documents/profile/test3 test3}
test3
rpc error: code = PermissionDenied desc = Missing or insufficient permissions.
rpc error: code = PermissionDenied desc = Missing or insufficient permissions.
rpc error: code = PermissionDenied desc = Missing or insufficient permissions.
rpc error: code = PermissionDenied desc = Missing or insufficient permissions.
Firestore rules - I opened up my firestore rules thinking that was the issue.
service cloud.firestore {
match /databases/{database}/documents {
match /{document=**} {
allow read, write;
}
}
}
I was able to get this work by changing the following configuration.
Go to Project Overview -> Project Settings -> Service accounts -> "Firebase Admin SDK" -> "Generate new private key" -> "Generate"
Then you see a popup like below
It took me an hour to figure this out. I hope this saves all your time!
I had the same problem, It's sad documentation only provides the most difficult ways to do it, as beginner it's important to make as easy as posible at first, but anyway these are the easiest steps to do it:
If you already have log in the console, jump to step 5
Intall gcloud
Run on your command line gcloud auth login
Login with the google account that owns the project
In your code, your configuration client should be like this:
ctx := context.Background()
client, err := firestore.NewClient(ctx, projectID)
if err != nil {
log.Fatal(err)
}
Run on your command line gcloud beta auth application-default login
And that's it! it's like magic, I had too much headache trying the other ways.
This works for cloud storage too and any google cloud api I'll guess
About your code:
client.Doc("profile/test3")
You are not getting any document at all there, it's just creat a *DocumentRef, in order to get the document you should do:
ctx:=context.Background()
snapShot,err:=client.Doc("profile/test3").Get(ctx)
if err!=nil{
log.Println(err)
return err
}
var myStruct myStructType
if err=snapShot.DataTo(&myStruct);err!=nil{
log.Println(err)
return err
}
Read documentation for more info
The docs tell me that Doc() returns a *DocumentRef, a reference to the document. To fetch the document itself, you should call Get() on it, which gives you a *DocumentSnapshot, and finally call Data() on that.
ref := client.Doc("profile/test3")
snap, err := ref.Get(ctx)
// FIXME error checking
data := snap.Data()
I want to create a service on kubernetes which manages helm charts on the cluster. It installs charts from a private chart repository. Since I didn't find any documents on how to use helm client api, I was looking for some samples or guidelines for creating a service on top of helm client.
FOR HELM3
As other answers pointed, with Helm 2, you need to talk with tiller which complicates stuff.
It is way more clean with Helm 3 since tiller was removed and helm client directly communicates with Kubernetes API Server.
Here is an example code to install a helm chart programmatically with helm3:
package main
import (
"fmt"
"os"
"helm.sh/helm/v3/pkg/action"
"helm.sh/helm/v3/pkg/chart/loader"
"helm.sh/helm/v3/pkg/kube"
_ "k8s.io/client-go/plugin/pkg/client/auth"
)
func main() {
chartPath := "/tmp/my-chart-0.1.0.tgz"
chart, err := loader.Load(chartPath)
if err != nil {
panic(err)
}
kubeconfigPath := "/tmp/my-kubeconfig"
releaseName := "my-release"
releaseNamespace := "default"
actionConfig := new(action.Configuration)
if err := actionConfig.Init(kube.GetConfig(kubeconfigPath, "", releaseNamespace), releaseNamespace, os.Getenv("HELM_DRIVER"), func(format string, v ...interface{}) {
fmt.Sprintf(format, v)
}); err != nil {
panic(err)
}
iCli := action.NewInstall(actionConfig)
iCli.Namespace = releaseNamespace
iCli.ReleaseName = releaseName
rel, err := iCli.Run(chart, nil)
if err != nil {
panic(err)
}
fmt.Println("Successfully installed release: ", rel.Name)
}
Since it took me some time to get this working here is a minimal example (no error handling, left details about kube config, ...) for listing release names:
package main
import (
"k8s.io/client-go/kubernetes"
"k8s.io/helm/pkg/helm"
"k8s.io/helm/pkg/helm/portforwarder"
)
func main() {
// omit getting kubeConfig, see: https://github.com/kubernetes/client-go/tree/master/examples
// get kubernetes client
client, _ := kubernetes.NewForConfig(kubeConfig)
// port forward tiller
tillerTunnel, _ := portforwarder.New("kube-system", client, config)
// new helm client
helmClient := helm.NewClient(helm.Host(host))
// list/print releases
resp, _ := helmClient.ListReleases()
for _, release := range resp.Releases {
fmt.Println(release.GetName())
}
}
I was long trying to set up Helm installation with --set values, and I found that the best place to look currently available functionality is official helm documentation example and official Go docs for the client.
This only pertains to Helm 3.
Here's an example I managed to get working by using the resources linked above.
I haven't found a more elegant way to define values rather than recursively asking for the map[string]interface{}, so if anyone knows a better way, please let me know.
Values should be roughly equivalent to:
helm install myrelease /mypath --set redis.sentinel.masterName=BigMaster,redis.sentinel.pass="random" ... etc
Notice the use of settings.RESTClientGetter(), rather than kube.Get, as in other answers. I found kube.Get to be causing nasty conflicts with k8s clients.
package main
import (
"log"
"os"
"helm.sh/helm/v3/pkg/action"
"helm.sh/helm/v3/pkg/chart/loader"
"helm.sh/helm/v3/pkg/cli"
"helm.sh/helm/v3/pkg/release"
)
func main(){
chartPath := "/mypath"
namespace := "default"
releaseName := "myrelease"
settings := cli.New()
actionConfig := new(action.Configuration)
// You can pass an empty string instead of settings.Namespace() to list
// all namespaces
if err := actionConfig.Init(settings.RESTClientGetter(), namespace,
os.Getenv("HELM_DRIVER"), log.Printf); err != nil {
log.Printf("%+v", err)
os.Exit(1)
}
// define values
vals := map[string]interface{}{
"redis": map[string]interface{}{
"sentinel": map[string]interface{}{
"masterName": "BigMaster",
"pass": "random",
"addr": "localhost",
"port": "26379",
},
},
}
// load chart from the path
chart, err := loader.Load(chartPath)
if err != nil {
panic(err)
}
client := action.NewInstall(actionConfig)
client.Namespace = namespace
client.ReleaseName = releaseName
// client.DryRun = true - very handy!
// install the chart here
rel, err := client.Run(chart, vals)
if err != nil {
panic(err)
}
log.Printf("Installed Chart from path: %s in namespace: %s\n", rel.Name, rel.Namespace)
// this will confirm the values set during installation
log.Println(rel.Config)
}
I was looking for the same answer, since I do know the solution now, sharing it here.
What you are looking for is to write a wrapper around helm library.
First you need a client which speaks to the tiller of your cluster. For that you need to create a tunnel to the tiller from your localhost. Use this (its the same link as kiran shared.)
Setup the Helm environement variables look here
Use this next. It will return a helm client. (you might need to write a wrapper around it to work with your setup of clusters)
After you get the *helm.Client handle, you can use helm's client API given here. You just have to use the one you need with the appropriate values.
You might need some utility functions defined here, like loading a chart as a folder/archive/file.
If you want to do something more, you pretty much locate the method in the doc and call it using the client.