Dynamic Discovery Client For GCP Golang - go

I've recently shifted from python to golang. I had been using python to work with GCP.
I used to pass in the scopes and mention the discovery client I wanted to create like this :
def get_client(scopes, api, version="v1"):
service_account_json = os.environ.get("SERVICE_ACCOUNT_KEY_JSON", None)
if service_account_json is None:
sys.exit("Exiting !!! No SSH_KEY_SERVICE_ACCOUNT env var found.")
credentials = service_account.Credentials.from_service_account_info(
json.loads(b64decode(service_account_json)), scopes=scopes
)
return discovery.build(api, version, credentials=credentials, cache_discovery=False)
And this would create my desired discovery client, whether it be compute engine service or sqladmin
However in go I don't seem to find this.
I found this : https://pkg.go.dev/google.golang.org/api/discovery/v1
For any client that I want to create I would've to import that and then create that, like this :
https://cloud.google.com/resource-manager/reference/rest/v1/projects/list#examples
package main
import (
"fmt"
"log"
"golang.org/x/net/context"
"golang.org/x/oauth2/google"
"google.golang.org/api/cloudresourcemanager/v1"
)
func main() {
ctx := context.Background()
c, err := google.DefaultClient(ctx, cloudresourcemanager.CloudPlatformScope)
if err != nil {
log.Fatal(err)
}
cloudresourcemanagerService, err := cloudresourcemanager.New(c)
if err != nil {
log.Fatal(err)
}
req := cloudresourcemanagerService.Projects.List()
if err := req.Pages(ctx, func(page *cloudresourcemanager.ListProjectsResponse) error {
for _, project := range page.Projects {
// TODO: Change code below to process each `project` resource:
fmt.Printf("%#v\n", project)
}
return nil
}); err != nil {
log.Fatal(err)
}
}
So I've to import each client library to get the client for that.
"google.golang.org/api/cloudresourcemanager/v1"
There's no dynamic creation of it.
Is it even possible, cause go is strict type checking 🤔
Thanks.

No, this is not possible with the Golang Google Cloud library.
You've nailed the point on the strict type checking, as it would definitely defeat the benefits of compile time type checking. It would also be a bad Golang practice to return different objects with different signatures, as we don't do duck typing and instead we rely on interface contracts.
Golang is boring and verbose, and it's like that by design :)

Related

trying to unmarshal aws codepipeline result

I am very new at Golang AWS SDK V2, I had similar code work without AWS results, but this one is getting me issues since the types are different. Also, I have search and none of the examples is with the code pipeline aws-sdk-v2 with the type of JSON I have to unmarshal.
I hope some of you can help me.
— This below is main.go I have the structs in another file called un-marshal.go that I created with the result of getting the same output I need from awscli tool and passing it by https://mholt.github.io/json-to-go/
package main
import (
"context"
"fmt"
"log"
"encoding/json"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/service/codepipeline"
)
func main() {
pipeline_name := "frontend"
// Load the Shared AWS Configuration (~/.aws/config)
cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithRegion("eu-central-1"))
if err != nil {
log.Fatal(err)
}
if err != nil {
log.Fatalf("unable to load SDK config, %v", err)
}
client := codepipeline.NewFromConfig(cfg)
pipeJson, err := client.GetPipelineState(context.TODO(), &codepipeline.GetPipelineStateInput{
Name: &pipeline_name,
})
if err != nil {
log.Println("Error getting Pipeline")
}
var cookie PipeLineResult
json.Unmarshal(pipeJson, &cookie)
fmt.Println("The name of the pipeline is: %s",cookie.PipelineName)
}
The error I am getting is:
/main.go:39:17: cannot use pipeJson (variable of type *codepipeline.GetPipelineStateOutput) as type []byte in argument to json.Unmarshal
Here I am lost because it is a new type, and not sure if I should convert, how to convert or work with the native type etc.
Thanks in advance.

How to set scope for token in Google Play Developer Reporting API

I started to use Google Play Developer Reporting API with using Golang(https://pkg.go.dev/google.golang.org/api#v0.79.0/playdeveloperreporting/v1beta1), and faced with issue related to API scopes:
Code:
package main
import (
"context"
"encoding/json"
"fmt"
"golang.org/x/oauth2"
"golang.org/x/oauth2/google"
"golang.org/x/oauth2/jwt"
"google.golang.org/api/option"
"google.golang.org/api/playdeveloperreporting/v1beta1"
"io/ioutil"
)
const (
GoogleApplicationCredentials = "path_to_service_account_credentials"
ProjectID = "apps/{project_id}"
)
func main() {
tokenSource, err := getTokenSource(GoogleApplicationCredentials)
if err !=nil {
panic(err)
}
if err := getAnomalies(tokenSource, ProjectID); err != nil {
panic(err)
}
}
func getAnomalies(tokenSource oauth2.TokenSource, projectID string) error {
ctx := context.Background()
service, err := playdeveloperreporting.NewService(ctx, option.WithTokenSource(tokenSource))
if err != nil {
return err
}
anomaliesCall := service.Anomalies.List(projectID)
result, err := anomaliesCall.Do()
if err != nil {
return err
}
fmt.Printf("\nStatus: %d", result.HTTPStatusCode)
return nil
}
func getTokenSource(credentialFile string) (oauth2.TokenSource, error) {
ctx := context.Background()
b, err := ioutil.ReadFile(credentialFile)
if err != nil {
return nil, err
}
var c = struct {
Email string `json:"client_email"`
PrivateKey string `json:"private_key"`
}{}
if err := json.Unmarshal(b, &c); err != nil {
return nil, err
}
fmt.Printf("\nClient email: %s\n", c.Email)
config := &jwt.Config{
Email: c.Email,
PrivateKey: []byte(c.PrivateKey),
Scopes: []string{
"?????????",
},
TokenURL: google.JWTTokenURL,
}
return config.TokenSource(ctx), nil
}
My question is what scope I need to use? I didn't find in - https://developers.google.com/identity/protocols/oauth2/scopes
Thanks
When you access an api in this case Google Play Developer Reporting API most of the data is private user data. In order to access private user data your application needs the permissions of the owner of that data or someone with access.
To get that access we use Oauth2, not all methods are created equal depending upon which method within the Google Play Developer Reporting API you are trying to use will dictate which scope of access the user will need to grant you.
The easest way to tell which scope is to check the documentation
Lets look at the anomalies.list method for an example. If we scroll down to the bottom it tells you exactly which scope your user needs to authorize.
After a bit of checking i think there is only one scope for this api it is https://www.googleapis.com/auth/playdeveloperreporting. So if you request that scope then you should have access to the full api.
I had the same problem
scope
https://www.googleapis.com/auth/playdeveloperreporting
You are receiving this error either because your input OAuth2 scope name is invalid or it refers to a newer scope that is outside the domain of this legacy API.
This API was built at a time when the scope name format was not yet standardized. This is no longer the case and all valid scope names (both old and new) are catalogued at https://developers.google.com/identity/protocols/oauth2/scopes. Use that webpage to lookup (manually) the scope name associated with the API you are trying to call and use it to craft your OAuth2 request.

How can I debug my Golang API code to show me what is wrong?

I have this module that use Google Cloud API to retrieve a list of all running Virtual Machine instances for a particular project. I'm new to Go, and followed the intro tutorial to help me out. I'm still trying to debug my code but no luck.
The problem is I'm able to communicate to Google Cloud API and pass authentication but that is all I can get through
compute.go module:
compute.go is able to communicate to Google Cloud servers and pass authentication (I'm not getting an auth error)
// Copyright 2021 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package compute
// [START compute_instances_list_all]
import (
"context"
"fmt"
"io"
compute "cloud.google.com/go/compute/apiv1"
"google.golang.org/api/iterator"
computepb "google.golang.org/genproto/googleapis/cloud/compute/v1"
"google.golang.org/protobuf/proto"
)
// listAllInstances prints all instances present in a project, grouped by their zone.
func ListAllInstances(w io.Writer, projectID string) error {
// projectID := "your_project_id"
ctx := context.Background()
instancesClient, err := compute.NewInstancesRESTClient(ctx)
// instancesClient, err := compute.NewInstancesRESTClient(ctx, option.WithCredentialsFile(`C:\path\to\jsonkey.json`))
if err != nil {
return fmt.Errorf("NewInstancesRESTClient: %v", err)
}
defer instancesClient.Close()
// Use the `MaxResults` parameter to limit the number of results that the API returns per response page.
req := &computepb.AggregatedListInstancesRequest{
Project: projectID,
MaxResults: proto.Uint32(6),
}
it := instancesClient.AggregatedList(ctx, req)
fmt.Fprintf(w, "Instances found:\n")
// Despite using the `MaxResults` parameter, you don't need to handle the pagination
// yourself. The returned iterator object handles pagination
// automatically, returning separated pages as you iterate over the results.
for {
pair, err := it.Next()
if err == iterator.Done {
break
}
if err != nil {
return err
}
instances := pair.Value.Instances
if len(instances) > 0 {
fmt.Fprintf(w, "%s\n", pair.Key)
for _, instance := range instances {
fmt.Fprintf(w, "- %s %s\n", instance.GetName(), instance.GetMachineType())
}
}
}
return nil
}
// [END compute_instances_list_all]
However the problem is when I run my main function that calls ListAllInstances, it returns a <nil>. Not allowing me to know what is wrong.
caller api.go module where I run go run .:
package main
import (
"fmt"
"example.com/compute"
"bytes"
)
func main() {
buf := new(bytes.Buffer)
// Get a message and print it.
respone := compute.ListAllInstances(buf, "project-unique-id")
fmt.Println(respone)
}
How else can I further debug this to figure out what is wrong with my code?
You're not printing buf. Your function returns an object of type error, which is nil (no error!), the actual output is written to buf.
Either print it out:
func main() {
buf := new(bytes.Buffer)
// Get a message and print it.
err := compute.ListAllInstances(buf, "project-unique-id")
if err != nil {
panic(err)
}
fmt.Println(buf.String()) // <======= Print buf contents!
}
Or just use os.Stdout:
func main() {
err := compute.ListAllInstances(os.Stdout, "project-unique-id")
if err != nil {
panic(err)
}
}
To answer your question about debugging, try using VSCode with the Go extension, in there you can run a debugger, set breakpoints and step through the code line-by-line, watching how variables change.
See also Debug Go programs in VS Code.

Can't Figure out how to use GORM's DBResolver

import (
"github.com/jinzhu/gorm"
_ "github.com/jinzhu/gorm/dialects/mysql"
"server/config"
"gorm.io/plugin/dbresolver"
)
func DB(config *config.Config) {
var err error
config.DB, err = gorm.Open("mysql", config.DBDSN)
if err != nil {
panic(err)
}
if !config.IsDev {
config.DB.Use(dbresolver.Register(dbresolver.Config{
Replicas: []gorm.Dialector{mysql.Open("mysql", config.DBDSN2)},
}))
}
}
I am trying to use GORM's DBresolver to make use of my cloud SQL instance's read replica. I think there is some versioning issue with the GORM package that I use and the DBresolver plugin. When I run the code above I get the following error:
config.DB.Use undefined (type *"github.com/jinzhu/gorm".DB has no
field or method Use)
undefined: "github.com/jinzhu/gorm".Dialect
I could not find any reason for this online and there are very few online resources on GORM's advanced functionalities. I can easily get the replica to work if I create a separate connection to it, but that way I need to specify the DB every-time I interact with the database.
I used the following to implement the code above: https://gorm.io/docs/dbresolver.html
You are using the v1 import path for gorm, but DBResolver is a V2 feature.
You'll want instead to use:
"gorm.io/gorm" for the main import package
"gorm.io/driver/mysql" for the driver import
gorm.Open with mysql.Open for creating the connections.
V2 is mostly backwards compatible, but you'll need to make sure any old code is tested to work on the new version.
import (
"gorm.io/gorm"
"gorm.io/driver/mysql"
"gorm.io/plugin/dbresolver"
"server/config"
)
func DB(config *config.Config) {
var err error
config.DB, err := gorm.Open(mysql.Open(config.DBDSN), &gorm.Config{})
if err != nil {
panic(err)
}
if !config.IsDev {
config.DB.Use(dbresolver.Register(dbresolver.Config{
Replicas: []gorm.Dialector{mysql.Open("mysql", config.DBDSN2)},
}))
}
}

Get current resource usage of a pod in Kubernetes with Go client

The kubernetes go client has tons of methods and I can't find how I can get the current CPU & RAM usage of a specific (or all pods).
Can someone tell me what methods I need to call to get the current usage for pods & nodes?
My NodeList:
nodes, err := clientset.CoreV1().Nodes().List(metav1.ListOptions{})
Kubernetes Go Client: https://github.com/kubernetes/client-go
Metrics package: https://github.com/kubernetes/kubernetes/tree/master/staging/src/k8s.io/metrics
As far as I got the metrics server implements the Kubernetes metrics package in order to fetch the resource usage from pods and nodes, but I couldn't figure out where & how they do it: https://github.com/kubernetes-incubator/metrics-server
It is correct that go-client does not have support for metrics type, but in the metrics package there is a pregenerated client that can be used for fetching metrics objects and assign them right away to the appropriate structure. The only thing you need to do first is to generate a config and pass it to metrics client. So a simple client for metrics would look like this:
package main
import (
"k8s.io/client-go/tools/clientcmd"
metrics "k8s.io/metrics/pkg/client/clientset/versioned"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
func main() {
var kubeconfig, master string //empty, assuming inClusterConfig
config, err := clientcmd.BuildConfigFromFlags(master, kubeconfig)
if err != nil{
panic(err)
}
mc, err := metrics.NewForConfig(config)
if err != nil {
panic(err)
}
mc.MetricsV1beta1().NodeMetricses().Get("your node name", metav1.GetOptions{})
mc.MetricsV1beta1().NodeMetricses().List(metav1.ListOptions{})
mc.MetricsV1beta1().PodMetricses(metav1.NamespaceAll).List(metav1.ListOptions{})
mc.MetricsV1beta1().PodMetricses(metav1.NamespaceAll).Get("your pod name", metav1.GetOptions{})
}
Each of the above methods from metric client returns an appropriate structure (you can check those here) and an error (if any) which you should process according to your requirements.
here is an example.
package main
import (
"fmt"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/tools/clientcmd"
metrics "k8s.io/metrics/pkg/client/clientset/versioned"
)
func main() {
var kubeconfig, master string //empty, assuming inClusterConfig
config, err := clientcmd.BuildConfigFromFlags(master, kubeconfig)
if err != nil {
panic(err)
}
mc, err := metrics.NewForConfig(config)
if err != nil {
panic(err)
}
podMetrics, err := mc.MetricsV1beta1().PodMetricses(metav1.NamespaceAll).List(metav1.ListOptions{})
if err != nil {
fmt.Println("Error:", err)
return
}
for _, podMetric := range podMetrics.Items {
podContainers := podMetric.Containers
for _, container := range podContainers {
cpuQuantity, ok := container.Usage.Cpu().AsInt64()
memQuantity, ok := container.Usage.Memory().AsInt64()
if !ok {
return
}
msg := fmt.Sprintf("Container Name: %s \n CPU usage: %d \n Memory usage: %d", container.Name, cpuQuantity, memQuantity)
fmt.Println(msg)
}
}
}
The API you're looking for in new versions of Kubernetes (tested on mine as of 1.10.7) is the metrics.k8s.io/v1beta1 API route.
You can see it locally if you run a kubectl proxy and check http://localhost:8001/apis/metrics.k8s.io/v1beta1/pods and /nodes on your localhost.
I see where your confusion is though. At the time of writing, it does not look like the metrics/v1beta1 has a generated typed package (https://godoc.org/k8s.io/client-go/kubernetes/typed), and doesn't appear in the kubernetes.ClientSet object.
You can hit all available endpoints directly though the rest.RestClient object, and just specify metrics/v1beta1 as the versionedAPIPath, which will be more work and less convenient than the nicely wrapped ClientSet, but I'm not sure how long it'll take before that API shows up in that interface.

Resources