Batch requests to GCP Compute using Golang client library - go

Is it possible to perform batch requests using the Google API Client Library for Go?
More precisely, I'd like to delete some disks; it would be great if I could avoid a request for each disk.
The batch request feature I'm referring to:
https://cloud.google.com/compute/docs/api/how-tos/batch
The client library I'm using:
https://github.com/googleapis/google-api-go-client
It seems there's also another client library, but without support for the Compute Engine, only metadata (?):
https://github.com/GoogleCloudPlatform/google-cloud-go

I'm not sure but i think it can be done since we can dlete a disk using API call.
A batch request consists of multiple API calls combined into one HTTP request, which can be sent to the batchPath specified in the API discovery document. The default path is /batch/api_name/api_version.
Example take a look on the Format of a batch request Here
// method id "compute.disks.delete":
type DisksDeleteCall struct {
s *Service
project string
zone string
disk string
urlParams_ gensupport.URLParams
ctx_ context.Context
header_ http.Header
}
// Delete: Deletes the specified persistent disk. Deleting a disk
// removes its data permanently and is irreversible. However, deleting a
// disk does not delete any snapshots previously made from the disk. You
// must separately delete snapshots.
// For details, see https://cloud.google.com/compute/docs/reference/latest/disks/delete
func (r *DisksService) Delete(project string, zone string, disk string) *DisksDeleteCall {
c := &DisksDeleteCall{s: r.s, urlParams_: make(gensupport.URLParams)}
c.project = project
c.zone = zone
c.disk = disk
return c
}
// RequestId sets the optional parameter "requestId": An optional
// request ID to identify requests. Specify a unique request ID so that
// if you must retry your request, the server will know to ignore the
// request if it has already been completed.
//
// For example, consider a situation where you make an initial request
// and the request times out. If you make the request again with the
// same request ID, the server can check if original operation with the
// same request ID was received, and if so, will ignore the second
// request. This prevents clients from accidentally creating duplicate
// commitments.
//
// The request ID must be a valid UUID with the exception that zero UUID
// is not supported (00000000-0000-0000-0000-000000000000).
func (c *DisksDeleteCall) RequestId(requestId string) *DisksDeleteCall {
c.urlParams_.Set("requestId", requestId)
return c
}
// Fields allows partial responses to be retrieved. See
// https://developers.google.com/gdata/docs/2.0/basics#PartialResponse
// for more information.
func (c *DisksDeleteCall) Fields(s ...googleapi.Field) *DisksDeleteCall {
c.urlParams_.Set("fields", googleapi.CombineFields(s))
return c
}
// Context sets the context to be used in this call's Do method. Any
// pending HTTP request will be aborted if the provided context is
// canceled.
func (c *DisksDeleteCall) Context(ctx context.Context) *DisksDeleteCall {
c.ctx_ = ctx
return c
}
// Header returns an http.Header that can be modified by the caller to
// add HTTP headers to the request.
func (c *DisksDeleteCall) Header() http.Header {
if c.header_ == nil {
c.header_ = make(http.Header)
}
return c.header_
}
func (c *DisksDeleteCall) doRequest(alt string) (*http.Response, error) {
reqHeaders := make(http.Header)
for k, v := range c.header_ {
reqHeaders[k] = v
}
reqHeaders.Set("User-Agent", c.s.userAgent())
var body io.Reader = nil
c.urlParams_.Set("alt", alt)
c.urlParams_.Set("prettyPrint", "false")
urls := googleapi.ResolveRelative(c.s.BasePath, "{project}/zones/{zone}/disks/{disk}")
urls += "?" + c.urlParams_.Encode()
req, _ := http.NewRequest("DELETE", urls, body)
req.Header = reqHeaders
googleapi.Expand(req.URL, map[string]string{
"project": c.project,
"zone": c.zone,
"disk": c.disk,
})
return gensupport.SendRequest(c.ctx_, c.s.client, req)
}
// Do executes the "compute.disks.delete" call.
// Exactly one of *Operation or error will be non-nil. Any non-2xx
// status code is an error. Response headers are in either
// *Operation.ServerResponse.Header or (if a response was returned at
// all) in error.(*googleapi.Error).Header. Use googleapi.IsNotModified
// to check whether the returned error was because
// http.StatusNotModified was returned.
func (c *DisksDeleteCall) Do(opts ...googleapi.CallOption) (*Operation, error) {
gensupport.SetOptions(c.urlParams_, opts...)
res, err := c.doRequest("json")
if res != nil && res.StatusCode == http.StatusNotModified {
if res.Body != nil {
res.Body.Close()
}
return nil, &googleapi.Error{
Code: res.StatusCode,
Header: res.Header,
}
}
if err != nil {
return nil, err
}
defer googleapi.CloseBody(res)
if err := googleapi.CheckResponse(res); err != nil {
return nil, err
}
ret := &Operation{
ServerResponse: googleapi.ServerResponse{
Header: res.Header,
HTTPStatusCode: res.StatusCode,
},
}
target := &ret
if err := gensupport.DecodeResponse(target, res); err != nil {
return nil, err
}
return ret, nil
// {
// "description": "Deletes the specified persistent disk. Deleting a disk removes its data permanently and is irreversible. However, deleting a disk does not delete any snapshots previously made from the disk. You must separately delete snapshots.",
// "httpMethod": "DELETE",
// "id": "compute.disks.delete",
// "parameterOrder": [
// "project",
// "zone",
// "disk"
// ],
// "parameters": {
// "disk": {
// "description": "Name of the persistent disk to delete.",
// "location": "path",
// "required": true,
// "type": "string"
// },
// "project": {
// "description": "Project ID for this request.",
// "location": "path",
// "pattern": "(?:(?:[-a-z0-9]{1,63}\\.)*(?:[a-z](?:[-a-z0-9]{0,61}[a-z0-9])?):)?(?:[0-9]{1,19}|(?:[a-z0-9](?:[-a-z0-9]{0,61}[a-z0-9])?))",
// "required": true,
// "type": "string"
// },
// "requestId": {
// "description": "An optional request ID to identify requests. Specify a unique request ID so that if you must retry your request, the server will know to ignore the request if it has already been completed.\n\nFor example, consider a situation where you make an initial request and the request times out. If you make the request again with the same request ID, the server can check if original operation with the same request ID was received, and if so, will ignore the second request. This prevents clients from accidentally creating duplicate commitments.\n\nThe request ID must be a valid UUID with the exception that zero UUID is not supported (00000000-0000-0000-0000-000000000000).",
// "location": "query",
// "type": "string"
// },
// "zone": {
// "description": "The name of the zone for this request.",
// "location": "path",
// "pattern": "[a-z](?:[-a-z0-9]{0,61}[a-z0-9])?",
// "required": true,
// "type": "string"
// }
// },
// "path": "{project}/zones/{zone}/disks/{disk}",
// "response": {
// "$ref": "Operation"
// },
// "scopes": [
// "https://www.googleapis.com/auth/cloud-platform",
// "https://www.googleapis.com/auth/compute"
// ]
// }
}

Related

Unmarshal nested GRPC structure in go

We want to unmarshal (in golang) a GRPC message and transform it into a map[string]interface{} to further process it. After using this code:
err := ptypes.UnmarshalAny(resource, config)
configMarshal, err := json.Marshal(config)
var configInterface map[string]interface{}
err = json.Unmarshal(configMarshal, &configInterface)
we get the following structure:
{
"name": "envoy.filters.network.tcp_proxy",
"ConfigType": {
"TypedConfig": {
"type_url": "type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy",
"value": "ChBCbGFja0hvbGVDbHVzdGVyEhBCbGFja0hvbGVDbHVzdGVy"
}
}
}
Where the TypedConfig field remains encoded. How can we decode the TypedConfig field? We know the type_url and we know the value, but to unmarshal the field, it needs to be of the pbany.Any type. But because the TypedConfig structure is a map[string] interface {}, our program either fails to compile, or it crashes, complaining that it is expecting a pbany.Any type, but instead it is getting a map[string] interface {}.
We have the following questions:
Is there a way to turn the structure under TypedConfig into a pbany.Any type that can be subsequently unmarshalled?
Is there a way to recursively unmarshal the entire GRPC message?
Edit (provide more information about the code, schemas/packages used)
We are looking at the code of xds_proxy.go here: https://github.com/istio/istio/blob/master/pkg/istio-agent/xds_proxy.go
This code uses a *discovery.DiscoveryResponse structure in this function:
func forwardToEnvoy(con *ProxyConnection, resp *discovery.DiscoveryResponse) {
The protobuf schema for discovery.DiscoveryResponse (and every other structure used in the code) is in the https://github.com/envoyproxy/go-control-plane/ repository in this file: https://github.com/envoyproxy/go-control-plane/blob/main/envoy/service/discovery/v3/discovery.pb.go
We added code to the forwardToEnvoy function to see the entire unmarshalled contents of the *discovery.DiscoveryResponse structure:
var config proto.Message
switch resp.TypeUrl {
case "type.googleapis.com/envoy.config.route.v3.RouteConfiguration":
config = &route.RouteConfiguration{}
case "type.googleapis.com/envoy.config.listener.v3.Listener":
config = &listener.Listener{}
// Six more cases here, truncated to save space
}
for _, resource := range resp.Resources {
err := ptypes.UnmarshalAny(resource, config)
if err != nil {
proxyLog.Infof("UnmarshalAny err %v", err)
return false
}
configMarshal, err := json.Marshal(config)
if err != nil {
proxyLog.Infof("Marshal err %v", err)
return false
}
var configInterface map[string]interface{}
err = json.Unmarshal(configMarshal, &configInterface)
if err != nil {
proxyLog.Infof("Unmarshal err %v", err)
return false
}
}
And this works well, except that now we have these TypedConfig fields that are still encoded:
{
"name": "virtualOutbound",
"address": {
"Address": {
"SocketAddress": {
"address": "0.0.0.0",
"PortSpecifier": {
"PortValue": 15001
}
}
}
},
"filter_chains": [
{
"filter_chain_match": {
"destination_port": {
"value": 15001
}
},
"filters": [
{
"name": "istio.stats",
"ConfigType": {
"TypedConfig": {
"type_url": "type.googleapis.com/udpa.type.v1.TypedStruct",
"value": "CkF0eXBlLmdvb2dsZWFwaXMuY29tL2Vudm95LmV4dGVuc2lvbnMuZmlsdG"
}
}
},
One way to visualize the contents of the TypedConfig fields is to use this code:
for index1, filterChain := range listenerConfig.FilterChains {
for index2, filter := range filterChain.Filters {
proxyLog.Infof("Listener %d: Handling filter chain %d, filter %d", i, index1, index2)
switch filter.ConfigType.(type) {
case *listener.Filter_TypedConfig:
proxyLog.Infof("Found TypedConfig")
typedConfig := filter.GetTypedConfig()
proxyLog.Infof("typedConfig.TypeUrl = %s", typedConfig.TypeUrl)
switch typedConfig.TypeUrl {
case "type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy":
tcpProxyConfig := &tcp_proxy.TcpProxy{}
err := typedConfig.UnmarshalTo(tcpProxyConfig)
if err != nil {
proxyLog.Errorf("Failed to unmarshal TCP proxy configuration")
} else {
proxyLog.Infof("TcpProxy Config for filter chain %d filter %d: %s", index1, index2, prettyPrint(tcpProxyConfig))
}
But then the code becomes very complex, as we have a large number of structures, and these structures can occur in different order in the messages.
So we wanted to get a generic way of unmarshalling these TypedConfig message by using pbAny, and hence our questions.

Beego endpoint can't find templatefile... but i'm not using template

I'm having trouble to create a endpoint on a Beego application
So, I just put some object information on the returned JSON:
// GetOne ...
// #Title GetOne
// #Description get Migration by id
// #Param id path string true "The key for staticblock"
// #Success 200 {object} models.Migration
// #Failure 403 :id is empty
// #router /:id [get]
func (c *MigrationController) GetOne() {
val, err := mg.Data["json"] = map[string]string{
"MigrationId": c.MigrationId
"Status": c.Status
"Created": c.Created
"Updated": c.Updated
}
if err != nil {
log.Debug("Fail - GetOne: %v", err)
} else {
mg.ServeJSON()
}
When I tried to call the endpoint, I get this
Handler crashed with error can't find templatefile in the path:views/migrationcontroller/getone.tpl
I'm not using these templates anywhere in the whole code...
I'm not familiar with this framework, someone can help me?
You should use ServeJSON() with current controller.
func (c *MigrationController) GetOne() {
defer c.ServeJSON()
...
}

Golang - How to create callback notification

I am trying to implement a case where a client sends a POST request to subscribe to some service to a server.
The server has to respond with the subscription data, however after some time if there is a change in the subscription information in the server, the server has to send a notification to the client about the changes using the "nfStatusNotificationUri" in the request body ("nfStatusNotificationUri" in the JSON
data below).
I do not know how to do this.
I have implemented the POST subscription part but have no idea how to implement the notification part.
Can anyone help me or give me some guidance onenter code here how to do this.
This what I have done so far:
// server
// functions
func (m *NfInstanceDataAccess) Insertsub(nfinstancesub Subscriptions) error {
err := db.C(COLLECTION).Insert(&nfinstancesub)
return err
}
func CreateNewSubscriptionPost(w http.ResponseWriter, r *http.Request) {
var nfinstancesub Subscriptions
id := uuid.New()
subscriptionID := id.String()
if r.Header.Get("Accept") != "application/json" {
WriteError(w, ErrNotAcceptable)
return
}
if err := json.NewDecoder(r.Body).Decode(&nfinstancesub); err != nil {
respondWithError(w, http.StatusBadRequest, "Invalid request payload")
return
}
nfinstancesub.ID = bson.NewObjectId()
nfinstancesub.SubscriptionID = subscriptionID
if err := da.Insertsub(nfinstancesub); err != nil {
respondWithError(w, http.StatusInternalServerError, err.Error())
return
}
w.Header().Set("Response-Code", "201")
w.Header().Set("Response-Desc", "Success")
w.Header().Set("Cache-Control", "max-age=2592000") // 30 days
respondWithJson(w, http.StatusCreated, nfinstancesub)
}
// Main function
func main() {
http.HandleFunc("/nnrf-nfm/v1/subscriptions, CreateNewSubscriptionPost)
log.Fatal(http.ListenAndServe(":8080", nil))
}
The JSON data to request subscription including the notification uri "nfStatusNotificationUri" field.
Am using mongodb to store this JSON data request is sent.
{
"nfStatusNotificationUri": "string",
"subscriptionID": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"validityTime": "2019-02-11T09:45:52.015Z",
"reqNotifEvents": [
"NF_REGISTERED",
"string"
],
"plmnId": {
"mcc": "string",
"mnc": "string"
},
"notifCondition": {
"monitoredAttributes": [
"string"
],
"unmonitoredAttributes": [
"string"
]
},
"reqNfFqdn": "string"
}

how I can mapping complex JSON to other JSON

I am trying to build aggregation services, to all third party APIs that's I used,
this aggregation services taking json values coming from my main system and it will put this value to key equivalent to third party api key then, aggregation services it will send request to third party api with new json format.
example-1:
package main
import (
"encoding/json"
"fmt"
"log"
"github.com/tidwall/gjson"
)
func main() {
// mapping JSON
mapB := []byte(`
{
"date": "createdAt",
"clientName": "data.user.name"
}
`)
// from my main system
dataB := []byte(`
{
"createdAt": "2017-05-17T08:52:36.024Z",
"data": {
"user": {
"name": "xxx"
}
}
}
`)
mapJSON := make(map[string]interface{})
dataJSON := make(map[string]interface{})
newJSON := make(map[string]interface{})
err := json.Unmarshal(mapB, &mapJSON)
if err != nil {
log.Panic(err)
}
err = json.Unmarshal(dataB, &dataJSON)
if err != nil {
log.Panic(err)
}
for i := range mapJSON {
r := gjson.GetBytes(dataB, mapJSON[i].(string))
newJSON[i] = r.Value()
}
newB, err := json.MarshalIndent(newJSON, "", " ")
if err != nil {
log.Println(err)
}
fmt.Println(string(newB))
}
output:
{
"clientName": "xxx",
"date": "2017-05-17T08:52:36.024Z"
}
I use gjson package to get values form my main system request in simple way from a json document.
example-2:
import (
"encoding/json"
"fmt"
"log"
"github.com/tidwall/gjson"
)
func main() {
// mapping JSON
mapB := []byte(`
{
"date": "createdAt",
"clientName": "data.user.name",
"server":{
"google":{
"date" :"createdAt"
}
}
}
`)
// from my main system
dataB := []byte(`
{
"createdAt": "2017-05-17T08:52:36.024Z",
"data": {
"user": {
"name": "xxx"
}
}
}
`)
mapJSON := make(map[string]interface{})
dataJSON := make(map[string]interface{})
newJSON := make(map[string]interface{})
err := json.Unmarshal(mapB, &mapJSON)
if err != nil {
log.Panic(err)
}
err = json.Unmarshal(dataB, &dataJSON)
if err != nil {
log.Panic(err)
}
for i := range mapJSON {
r := gjson.GetBytes(dataB, mapJSON[i].(string))
newJSON[i] = r.Value()
}
newB, err := json.MarshalIndent(newJSON, "", " ")
if err != nil {
log.Println(err)
}
fmt.Println(string(newB))
}
output:
panic: interface conversion: interface {} is map[string]interface {}, not string
I can handle this error by using https://golang.org/ref/spec#Type_assertions, but what if this json object have array and inside this array have json object ....
my problem is I have different apis, every api have own json schema, and my way for mapping json only work if
third party api have json key value only, without nested json or array inside this array json object.
is there a way to mapping complex json schema, or golang package to help me to do that?
EDIT:
After comment interaction and with updated question. Before we move forward, I would like to mention.
I just looked at your example-2 Remember one thing. Mapping is from one form to another form. Basically one known format to targeted format. Each data type have to handled. You cannot do generic to generic mapping logically (technically feasible though, would take more time & efforts, you can play around on this).
I have created sample working program of one approach; it does a mapping of source to targeted format. Refer this program as a start point and use your creativity to implement yours.
Playground link: https://play.golang.org/p/MEk_nGcPjZ
Explanation: Sample program achieves two different source format to one target format. The program consist of -
Targeted Mapping definition of Provider 1
Targeted Mapping definition of Provider 2
Provider 1 JSON
Provider 2 JSON
Mapping function
Targeted JSON marshal
Key elements from program: refer play link for complete program.
type MappingInfo struct {
TargetKey string
SourceKeyPath string
DataType string
}
Map function:
func mapIt(mapping []*MappingInfo, parsedResult gjson.Result) map[string]interface{} {
mappedData := make(map[string]interface{})
for _, m := range mapping {
switch m.DataType {
case "time":
mappedData[m.TargetKey] = parsedResult.Get(m.SourceKeyPath).Time()
case "string":
mappedData[m.TargetKey] = parsedResult.Get(m.SourceKeyPath).String()
}
}
return mappedData
}
Output:
Provider 1 Result: map[date:2017-05-17 08:52:36.024 +0000 UTC clientName:provider1 username]
Provider 1 JSON: {
"clientName": "provider1 username",
"date": "2017-05-17T08:52:36.024Z"
}
Provider 2 Result: map[date:2017-05-12 06:32:46.014 +0000 UTC clientName:provider2 username]
Provider 2 JSON: {
"clientName": "provider2 username",
"date": "2017-05-12T06:32:46.014Z"
}
Good luck, happy coding!
Typically Converting/Transforming one structure to another structure, you will have to handle this with application logic.
As you mentioned in the question:
my problem is I have different apis, every api have own json schema
This is true for every aggregation system.
One approach to handle this requirement effectively; is to keep mapping of keys for each provider JSON structure and targeted JSON structure.
For example: This is an approach, please go with your design as you see fit.
JSON structures from various provider:
// Provider 1 : JSON structrure
{
"createdAt": "2017-05-17T08:52:36.024Z",
"data": {
"user": {
"name": "xxx"
}
}
}
// Provider 2 : JSON structrure
{
"username": "yyy"
"since": "2017-05-17T08:52:36.024Z",
}
Mapping for target JSON structure:
jsonMappingByProvider := make(map[string]string)
// Targeted Mapping for Provider 1
jsonMappingByProvider["provider1"] = `
{
"date": "createdAt",
"clientName": "data.user.name"
}
`
// Targeted Mapping for Provider 2
jsonMappingByProvider["provider2"] = `
{
"date": "since",
"clientName": "username"
}
`
Now, based the on the provider you're handling, get the mapping and map the response JSON into targeted structure.
// get the mapping info by provider
mapping := jsonMappingByProvider["provider1"]
// Parse the response JSON
// Do the mapping
This way you can control each provider and it's mapping effectively.

Using function to satisfy interface via method

I am trying to write a method that will return a function that can satisfy the json.Marshaler interface. My reasoning is to provide different representations of the struct. Perhaps I am approaching this completely wrong.
func (api *Api) SiteList(c *gin.Context) {
var sites []db.Site
if err := api.db.Find(&sites).Error; err != nil {
}
var payload []json.Marshaler
for _, site := range sites {
payload = append(payload, site.ToApi())
}
c.JSON(http.StatusOK, payload)
}
The result I get from this function is the correct number of items in the list, but the same value for each:
[
{
"key": "NZ7LCA9HQN3",
"name": "autumn-waterfall-1573"
},
{
"key": "NZ7LCA9HQN3",
"name": "autumn-waterfall-1573"
},
{
"key": "NZ7LCA9HQN3",
"name": "autumn-waterfall-1573"
},
{
"key": "NZ7LCA9HQN3",
"name": "autumn-waterfall-1573"
},
{
"key": "NZ7LCA9HQN3",
"name": "autumn-waterfall-1573"
}
]
Finally, here is the ToApi implementation:
type EncoderFunc func() ([]byte, error)
func (fn EncoderFunc) MarshalJSON() ([]byte, error) {
return fn()
}
func (site *Site) ToApi() json.Marshaler {
return EncoderFunc(func() ([]byte, error) {
var payload public.Site
payload.Name = site.Name
payload.Key = site.Key
data, err := json.Marshal(payload)
if err != nil {
return nil, err
}
return data, nil
})
}
This seems like a classic closure gotcha. There is a related FAQ section.
Basically, site in your for-loop has the same address every time. All your functions are closed over this address. So when you evaluate it after the for-loop, it will repeatedly call MarshalJSON on the value on the same (last) address. You can correct that by creating a new value on every iteration:
for _, site := range sites {
site := site // Perfectly legal and idiomatic.
payload = append(payload, site.ToApi())
}
Playground: https://play.golang.org/p/eFujC1hEyD
Another related piece of documentation from Effective Go:
The bug is that in a Go for loop, the loop variable is reused for each iteration, so the req variable is shared across all goroutines. That's not what we want. (...) Another solution is just to create a new variable with the same name, as in this example:
for req := range queue {
req := req // Create new instance of req for the goroutine.
sem <- 1
go func() {
process(req)
<-sem
}()
}
This section is about goroutines, but apparently this also applies to all closures.

Resources