I have a field type proto.Any passed from upstream service, and I need to convert it to proto.Struct. I see there is a UnmarshalAny function but it only takes proto.Message. Anybody can help
End up going with types.Any -> proto message -> jsonpb -> types.Struct
As Jochen mentioned in the comments, you can use anypb and structpb to manage the respective Well Known Types. So you will first import the followings:
"google.golang.org/protobuf/types/known/anypb"
"google.golang.org/protobuf/types/known/structpb"
and then it is basically just marshalling and unmarshalling process:
s := &structpb.Struct{
Fields: map[string]*structpb.Value{
"is_working": structpb.NewBoolValue(true),
},
}
any, err := anypb.New(s) // transform `s` to Any
if err != nil {
log.Fatalf("Error while creating Any from Struct")
}
m := new(structpb.Struct)
if err = any.UnmarshalTo(m); err != nil { // transform `any` back to Struct
log.Fatalf("Error while creating Struct from Any")
}
log.Println(m)
Note that I don't know you proto definition so here instead of doing the any.New marshalling part, you will replace that with the any you receive from your upstream service.
Related
When working with DynamoDB in Golang, if a call to query has more results, it will set LastEvaluatedKey on the QueryOutput, which you can then pass in to your next call to query as ExclusiveStartKey to pick up where you left off.
This works great when the values stay in Golang. However, I am writing a paginated API endpoint, so I would like to serialize this key so I can hand it back to the client as a pagination token. Something like this, where something is the magic package that does what I want:
type GetDomainObjectsResponse struct {
Items []MyDomainObject `json:"items"`
NextToken string `json:"next_token"`
}
func GetDomainObjects(w http.ResponseWriter, req *http.Request) {
// ... parse query params, set up dynamoIn ...
dynamoIn.ExclusiveStartKey = something.Decode(params.NextToken)
dynamoOut, _ := db.Query(dynamoIn)
response := GetDomainObjectsResponse{}
dynamodbattribute.UnmarshalListOfMaps(dynamoOut.Items, &response.Items)
response.NextToken := something.Encode(dynamoOut.LastEvaluatedKey)
// ... marshal and write the response ...
}
(please forgive any typos in the above, it's a toy version of the code I whipped up quickly to isolate the issue)
Because I'll need to support several endpoints with different search patterns, I would love a way to generate pagination tokens that doesn't depend on the specific search key.
The trouble is, I haven't found a clean and generic way to serialize the LastEvaluatedKey. You can marshal it directly to JSON (and then e.g. base64 encode it to get a token), but doing so is not reversible. LastEvaluatedKey is a map[string]types.AttributeValue, and types.AttributeValue is an interface, so while the json encoder can read it, it can't write it.
For example, the following code panics with panic: json: cannot unmarshal object into Go value of type types.AttributeValue.
lastEvaluatedKey := map[string]types.AttributeValue{
"year": &types.AttributeValueMemberN{Value: "1993"},
"title": &types.AttributeValueMemberS{Value: "Benny & Joon"},
}
bytes, err := json.Marshal(lastEvaluatedKey)
if err != nil {
panic(err)
}
decoded := map[string]types.AttributeValue{}
err = json.Unmarshal(bytes, &decoded)
if err != nil {
panic(err)
}
What I would love would be a way to use the DynamoDB-flavored JSON directly, like what you get when you run aws dynamodb query on the CLI. Unfortunately the golang SDK doesn't support this.
I suppose I could write my own serializer / deserializer for the AttributeValue types, but that's more effort than this project deserves.
Has anyone found a generic way to do this?
OK, I figured something out.
type GetDomainObjectsResponse struct {
Items []MyDomainObject `json:"items"`
NextToken string `json:"next_token"`
}
func GetDomainObjects(w http.ResponseWriter, req *http.Request) {
// ... parse query params, set up dynamoIn ...
eskMap := map[string]string{}
json.Unmarshal(params.NextToken, &eskMap)
esk, _ = dynamodbattribute.MarshalMap(eskMap)
dynamoIn.ExclusiveStartKey = esk
dynamoOut, _ := db.Query(dynamoIn)
response := GetDomainObjectsResponse{}
dynamodbattribute.UnmarshalListOfMaps(dynamoOut.Items, &response.Items)
lek := map[string]string{}
dynamodbattribute.UnmarshalMap(dynamoOut.LastEvaluatedKey, &lek)
response.NextToken := json.Marshal(lek)
// ... marshal and write the response ...
}
(again this is my real solution hastily transferred back to the toy problem, so please forgive any typos)
As #buraksurdar pointed out, attributevalue.Unmarshal takes an inteface{}. Turns out in addition to a concrete type, you can pass in a map[string]string, and it just works.
I believe this will NOT work if the AttributeValue is not flat, so this isn't a general solution [citation needed]. But my understanding is the LastEvaluatedKey returned from a call to Query will always be flat, so it works for this usecase.
Inspired by Dan, here is a solution to serialize and deserialize to/from base64
package dynamodb_helpers
import (
"encoding/base64"
"encoding/json"
"github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue"
"github.com/aws/aws-sdk-go-v2/service/dynamodb/types"
)
func Serialize(input map[string]types.AttributeValue) (*string, error) {
var inputMap map[string]interface{}
err := attributevalue.UnmarshalMap(input, &inputMap)
if err != nil {
return nil, err
}
bytesJSON, err := json.Marshal(inputMap)
if err != nil {
return nil, err
}
output := base64.StdEncoding.EncodeToString(bytesJSON)
return &output, nil
}
func Deserialize(input string) (map[string]types.AttributeValue, error) {
bytesJSON, err := base64.StdEncoding.DecodeString(input)
if err != nil {
return nil, err
}
outputJSON := map[string]interface{}{}
err = json.Unmarshal(bytesJSON, &outputJSON)
if err != nil {
return nil, err
}
return attributevalue.MarshalMap(outputJSON)
}
Background
I'm trying to analyze data from the Reddit api on users. I've declared a User struct like:
type User struct {
Kind string `json:"kind"`
Data struct {
...
Subreddit struct {
...
} `json:"subreddit"`
...
CreatedUtc float64 `json:"created_utc"` <---
...
} `json:"data"`
}
I request the data from the api and print it here.
func GetUser(url string) User {
var response User
resp, err := http.Get(url)
if err != nil {
...
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
...
}
err = json.Unmarshal(body, &response)
if err != nil {
...
}
fmt.Print(response.Data.CreatedUtc) <---
return response
}
Problem
When I request this endpoint it prints 0 while I can see in the browser that the created_utc timestamp is 1562538742. This seems to happen in the vast majority (but not all) cases.
Am I doing something wrong with my type conversions?
To understand why it is zero, you must first understand that in Go, types are not automatically references like in other languages. The variable var abc int will have a value of 0 by default.
When testing whether JSON is parsing correctly, you can change the values of the type to pointers. With this, any field that isn't filled is nil rather than the default value for that type.
Doing this, you can see if the value being returned is true, or if there is another failure, such as incorrect data model or failed network call.
Credit to #JimB for pointing out that I wasn't checking the status code of the response. I had expected that it would throw an error if it was > 400 but according to the docs that is not the case.
In my case, modifying the request to contain a user agent header resolved the issue.
I'm trying to build a generic function which will parse input (in JSON) into a specified structure. The structure may vary at run-time, based on parameters which are passed to the function. I'm currently trying to achieve this by passing an object of the right type and using reflect.New() to create a new output object of the same type.
I'm then parsing the JSON into this object, and scanning the fields.
If I create the object and specify the type in code, everything works. If I pass an object and try to create a replica, I get an "invalid indirect" error a few steps down (see code).
import (
"fmt"
"reflect"
"encoding/json"
"strings"
)
type Test struct {
FirstName *string `json:"FirstName"`
LastName *string `json:"LastName"`
}
func genericParser(incomingData *strings.Reader, inputStructure interface{}) (interface{}, error) {
//******* Use the line below and things work *******
//parsedInput := new(Test)
//******* Use vvv the line below and things don't work *******
parsedInput := reflect.New(reflect.TypeOf(inputStructure))
decoder := json.NewDecoder(incomingData)
err := decoder.Decode(&parsedInput)
if err != nil {
//parsing error
return nil, err
}
//******* This is the line that generates the error "invalid indirect of parsedInput (type reflect.Value)" *******
contentValues := reflect.ValueOf(*parsedInput)
for i := 0; i < contentValues.NumField(); i++ {
//do stuff with each field
fmt.Printf("Field name was: %s\n", reflect.TypeOf(parsedInput).Elem().Field(i).Name)
}
return parsedInput, nil
}
func main() {
inputData := strings.NewReader("{\"FirstName\":\"John\", \"LastName\":\"Smith\"}")
exampleObject := new(Test)
processedData, err := genericParser(inputData, exampleObject)
if err != nil {
fmt.Println("Parsing error")
} else {
fmt.Printf("Success: %v", processedData)
}
}
If I can't create a replica of the object, then a way of updating / returning the one supplied would be feasible. The key thing is that this function must be completely agnostic to the different structures available.
reflect.New isn't a direct analog to new, as it can't return a specific type, it only can return a reflect.Value. This means that you are attempting to unmarshal into a *reflect.Value, which obviously isn't going to work (even if it did, your code would have passed in **Type, which isn't what you want either).
Use parsedInput.Interface() to get the underlying value after creating the new value to unmarshal into. You then don't need to reflect on the same value a second time, as that would be a reflect.Value of a reflect.Value, which again isn't going to do anything useful.
Finally, you need to use parsedInput.Interface() before you return, otherwise you are returning the reflect.Value rather than the value of the input type.
For example:
func genericParser(incomingData io.Reader, inputStructure interface{}) (interface{}, error) {
parsedInput := reflect.New(reflect.TypeOf(inputStructure).Elem())
decoder := json.NewDecoder(incomingData)
err := decoder.Decode(parsedInput.Interface())
if err != nil {
return nil, err
}
for i := 0; i < parsedInput.Elem().NumField(); i++ {
fmt.Printf("Field name was: %s\n", parsedInput.Type().Elem().Field(i).Name)
}
return parsedInput.Interface(), nil
}
https://play.golang.org/p/CzDrj6sgQNt
TL;DR: How can I flexibly decode a k8s API object and inspect its top-level metav1.ObjectMeta struct without knowing the object's Kind in advance?
I'm writing an admission controller endpoint that unmarshals a metav1.AdmissionReview object's Request.Object.Raw field into a concrete object based on the Request.Kind field - e.g.
if kind == "Pod" {
var pod core.Pod
// ...
if _, _, err := deserializer.Decode(admissionReview.Request.Object.Raw, nil, &pod); err != nil {
return nil, err
}
annotations := pod.ObjectMeta.Annotations
// inspect/validate the annotations...
This requires knowing all possible types up front, or perhaps asking a user to supply a map[kind]corev1.Object that we can use to be more flexible.
What I'd like to instead achieve is something closer to:
var objMeta core.ObjectMeta
if _, _, err := deserializer.Decode(admissionReview.Request.Object.Raw, nil, &objMeta); err != nil {
return nil, err
}
// if objMeta is populated, validate the fields, else
// assume it is an object that does not define an ObjectMeta
// as part of its schema.
Is this possible? The k8s API surface is fairly extensive, and I've crawled through the metav1 godoc, corev1 godoc & https://cs.k8s.io for prior art without a decent example.
The closest I've found is possibly the ObjectMetaAccessor interface, but I'd need to get from an AdmissionReview.Request.Object (type runtime.RawExtension) to a runtime.Object first.
I believe you can't find what you are looking for because, when decoding an object, Kubernetes uses GetObjectKind and compares the result to a Scheme to convert the object to a concrete type, rather than using some generic like approach and interacting with the fields of an object without knowing it's concrete type.
So you can use reflection instead, something like:
k8sObjValue := reflect.ValueOf(admissionReview.Request.Object.Raw).Elem()
k8sObjObjectMeta := k8sObjValue.FieldByName("ObjectMeta")
annotations, ok := k8sObjObjectMeta.FieldByName("Annotations").Interface().(map[string]string)
if !ok {
panic("failed to retrieve annotations")
}
EDIT:
Or closer to your requirements, convert to an ObjectMeta object
k8sObjValue := reflect.ValueOf(admissionReview.Request.Object.Raw).Elem()
objMeta, ok := k8sObjValue.FieldByName("ObjectMeta").Interface().(core.ObjectMeta)
if !ok {
panic("failed to retrieve object metadata")
}
There is a way to do this that I discovered recently, let me describe it here:
Quick disclaimer: I used admission/v1 and never tested with admission/v1beta1, which should work identically.
The data type of admissionReview.Request.Object is runtime.RawExtension, and the k8s.io/apimachinery/pkg/runtime provides a method that can convert the runtime.RawExtension to a runtime.Object. The method is called runtime.Convert_runtime_RawExtension_To_runtime_Object(...). From there, you can easily convert to the unstructured.Unstructured data type, which has all the fields from the MetaV1 object accessible with simple getter methods.
Here is a code snippet that lets you get this accomplished:
import (
// ...
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
"k8s.io/apimachinery/pkg/runtime"
// ...
)
// ...
func dummyFunc(ar *v1.AdmissionReview) {
// ...
var obj runtime.Object
var scope conversion.Scope // While not actually used within the function, need to pass in
err := runtime.Convert_runtime_RawExtension_To_runtime_Object(&ar.Request.Object, &obj, scope)
if err != nil {
// ...
}
innerObj, err := runtime.DefaultUnstructuredConverter.ToUnstructured(obj)
if err != nil {
// ...
}
u := unstructured.Unstructured{Object: innerObj}
// Now the `u` variable has all the meta info available with simple getters.
// Sample:
labels := u.GetLabels()
kind := u.GetKind()
// etc.
// ...
}
References:
https://godoc.org/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured
https://godoc.org/k8s.io/apimachinery/pkg/runtime#Convert_runtime_RawExtension_To_runtime_Object
https://godoc.org/k8s.io/api/admission/v1#AdmissionRequest
It seems there are two possibilities:
Either the field Object should already contain the correct object instance, when using Go client, check code here.
Try using the converters here
while writing a golang webserver I had to use some sort of cache so i chose redis.
I had the need for some sort of function that takes any structure and saves it as is to redis as a value.
Is there any way to do this without using the interface{} as a receiving parameter or repeating myself too much and still staying type safe?
Encode the struct value to a []byte using the gob, json or similar encoding package. Store the []byte in Redis. Reverse the process when fetching the data.
Assuming a Redis client with methods for Set and Get, the code using the JSON package will look something like this:
func set(c *RedisClient, key string, value interface{}) error {
p, err := json.Marshal(value)
if err != nil {
return err
}
return c.Set(key, p)
}
func get(c *RedisClient, key string, dest interface{}) error {
p, err := c.Get(key)
if err != nil {
return err
}
return json.Unmarshal(p, dest)
}
Use it like this to save a value:
var v someType
if err := set(c, key, v); err != nil {
// handle error
}
and like this to retrieve a value. Note that a pointer to the value is passed to get.
var v someType
if err := get(c, key, &v); err != nil {
// handle error
}
The details will need to adjusted depending on the Redis client that you are using.
This approach avoids repetition and is type safe as long as the application sets and gets values for a given key using the same type.