Im trying to get RAW data from operator CR and im getting an empty object . (all others value are working as expected )
I’ve created a minimal example for the issue, in the example im trying to read the infrastructureConfig object
The tricky part here that my struct is reference to another struct which have a property type rawdata
https://github.com/JennyMet/gardner_test
Here the simple CR
https://github.com/JennyMet/gardner_test/blob/main/config/samples/mygroup_v1alpha1_rawtest.yaml#L11
Here im tying to read the data and get an empty object, any idea?
https://github.com/JennyMet/gardner_test/blob/main/controllers/rawtest_controller.go#L70
this is the reference of the type which im using
https://github.com/gardener/gardener/blob/5522be0e17ccf38aae36efb9fdb6463c66d6e4f1/pkg/apis/core/v1beta1/types_shoot.go#L1184
I think its related to fields
x-kubernetes-preserve-unknown-fields: true
https://book.kubebuilder.io/reference/markers/crd-processing.html
which is not existing
How can I add it to the schema here ?
https://github.com/JennyMet/gardner_test/blob/main/api/v1alpha1/rawtest_types.go#L32 as under the hood it uses
https://github.com/gardener/gardener/blob/5522be0e17ccf38aae36efb9fdb6463c66d6e4f1/pkg/apis/core/v1beta1/types_shoot.go#L1184
I mean I tried and it doesnt work as the InfrastructureConfig which is RAW is under the
type System struct {
Type system `json:"type,omitempty"`
// +kubebuilder:pruning:PreserveUnknownFields
Provider v1beta1.Provider `json:"provider,omitempty"`
}
But the rawData is under Provider which is not my struct, im just using it.
which is like this , see the InfrastructureConfig type...
type Provider struct {
Type string `json:"type" protobuf:"bytes,1,opt,name=type"`
ControlPlaneConfig *runtime.RawExtension `json:"controlPlaneConfig,omitempty" protobuf:"bytes,2,opt,name=controlPlaneConfig"`
InfrastructureConfig *runtime.RawExtension `json:"infrastructureConfig,omitempty" protobuf:"bytes,3,opt,name=infrastructureConfig"`
}
Currently, you can only put the // +kubebuilder:pruning:PreserveUnknownFields on the Provider v1beta1.Provider, which means all sub fields in it will be allowed with additional unknown fields.
The good news is, your problem will be solved after https://github.com/kubernetes-sigs/controller-tools/pull/683 merged. After that, you have not to use // +kubebuilder:pruning:PreserveUnknownFields and controller-tools would automatically add x-kubernetes-preserve-unknown-fields: true for all RawExtension fields.
Related
am using a protobuf definition file to create golang classes
file.proto
message my_message {
<what type to use?> data = 1 [json_name = "data"]
}
generated.go
type MyMessage struct {
data any
}
I have checked out Struct , Value and Any, though none of them actually mark the type as “any” in generated classes.
The closes I get is Value type which can accomodate primitives as well as structs and lists.
As far as I'm aware the Go code generator doesn't have any circumstances where any/interface{} will be used for a generated field.
google.protobuf.Any is likely what you want here, but the field in the generated code will of type anypb.Any. If you absolutely require the any type be used, you'll unfortunately have to manually map to another struct.
I am trying to do the following:
Deserialise a struct of fields into another struct of different format.
I am trying to do the transformation via an Hashmap as this seems to be the best suited way.
I am required to do this as a part of transformation of a legacy system, and this being one of the intermediary phases.
I have raised another question which caters to a subset of the same use-case, but do not think it gives enough overview, hence raising this question with more details and code.
(Rust converting a struct o key value pair where the field is not none)
Will be merging both questions once I figure out how.
Scenario:
I am getting input via an IPC through a huge proto.
I am using prost to deserialise this data into a struct.
My deserialised struct has n no of fields and can increase as well.
I need to transform this deserialised struct into a key,value struct. (shown ahead).
The incoming data, will mostly have a majority of null keys .i.e out of n fields, most likely only 1,2 have values at a given time.
Input struct after prost deserialisation:
Using proto3 so unable to define optional fields.
Ideally I would prefer a prost struct of options on every field. .i.e Option instead of string.
struct Data{
int32 field1,
int64 field2,
string field3,
...
}
This needs to be transformed to a genric struct as below for further use:
struct TransformedData
{
string Key
string Value
}
So,
Data{
field1: 20
field2: null
field3: null
field4: null
....
}
becomes
TransformedData{
key:"field1"
Value: "20"
}
Methods I have tried:
Method1
Add serde to the prost struct definiton and deserialise it into a map.
Loop over each item in a map to get values which are non-null.
Use these to create a struct
https://play.rust-lang.org/?version=stable&mode=debug&edition=2015&gist=9645b1158de31fd54976926c9665d6b4
This has its challenges:
Each primitve data type has its own default values and needs to be checked for.
Nested structs will result in object data type which needs to be handled.
Need to iterate over every field of a struct to determine non null values.
1&2 can be mitigated by setting an Optional field(yet to figure out how in proto3 and prost)
But I am not sure how to get over 3.
Iterating over a large struct to find non null fields is not scalable and will be expensive.
Method 2:
I am using prost reflects dynamic reflection to deserialise and get only specified value.
Doing this by ensuring each proto message has two extra fields:
proto -> signifying the proto struct being used when serializing.
key -> signifying the filed which has value when serializing.
let fd = package::FILE_DESCRIPTOR_SET;
let pool = DescriptorPool::decode(fd).unwrap();
let proto_message: package::GenericProto = prost::Message::decode(message.as_ref()).expect("error when de-serialising protobuf message to struct");
let proto = proto_message.proto.as_str();
let key = proto_message.key.as_str();
Then using key , I derive the key from what looks to be a map implemented by prost:
let message_descriptor = pool.get_message_by_name(proto).unwrap();
let dynamic_message = DynamicMessage::decode(message_descriptor, message.as_ref()).unwrap();
let data = dynamic_message.get_field_by_name(key).unwrap().as_ref().clone();
Here :
1.This fails when someone sends a struct with multiple fields filled.
2.This does not work for nested objects, as nested objects in turn needs to be converted to a map.
I am looking for the least expensive way to do the above.
I understand the solution I am looking for is pretty out there and I am taking a one size fits all approach.
Any pointers appreciated.
I have a standard response built in PHP for microservices that I want to replicate in Go.
In particular, the "data" element of the response array can have anything.
Example
$response['data'] = [];
$response['data']['entities'] = [ objects that implements toArray() ]
$response['data']['quantity'] = 1
Problem is that Data differs per endpoint.
So one option I was thinking is in each endpoint (controller, action, etc) declare a Data struct that I would use in the StandardResponse.Data struct.
But how can I achieve that? Using interfaces?
Sorry I am new to Go.
An interface{} is one way to do it. Something like:
type StandardResponse struct {
Error bool
ErrorMsg string
Data interface{}
}
Marshaling this to JSON will work assuming the underlying type of interface{} supports it.
For reference: example here
I am trying to access a struct field using a variable key, to explain in PHP you can do the below.
$arr = ["test" => "foo"];
$key = "test";
$result = $arr[$key];
Or in Javascript you can do the below
const obj = {"test": "foo"}
const key = "test"
const result = obj[key]
Is this possible with go structs? I have searched for this functionality, but the answers always seem to point to the reflect package and running a for loop over the struct fields.
My code (linked above) gets the compile error of invalid operation: p[key] (type Post does not support indexing) which makes sense, but I can't find a way around this.
One of the main points when using structs is that the way how to access the fields is known at compile time.Then the produced code uses fixed indexes added to a base address of the struct.
For any kind of dynamism here you just need to use map[string]string or similar.
I've been working on a package for API dev facilitation, that validates input data (according to a schema structure, mapping each field name to a value valdator/formator etc) ... I am sad to see that datastore does not want my payload=map[string]interface{} ...
I have then been playing a bit with the PropertyLoadSaver interface, constructing a slice of properties depending on a struct's values. All alues are pointers (which datastore does not accept excep struct), and I use it in order to not store the zero value for a non-provided value, but ignore it if the pointer is nil ...
It works pretty well, the problem come when I want to use embedded structs with pointer fields ... I thought I would just add a property with a name, and a value of type "entity" ... This entity has a nil key, and properties (the inner fields) ...
I thought this would let me handle a json POST binded to a struct like
Type Outer struct {
A *string
B *int
i *Inner
}
And
Type Inner struct {
C *bool
D *float64
}
... I would then fully use the power of nosql being schemaless and flexible, having entities that could have the optional property I or not, partially or totally filled (c and d could then also be optional) ...
This would be lighter, storing only the provided data and ignoring other properties (is the datastore GUI you can manually create entities of various forms) ... Retrieving lightweight struct with nil set to the pointers value while loading property if not set in the datastore entity retrieved from db, and thus not displaying to the user "bad" zero values, but not displaying at all what has not been provided and stored previously ...
In the gui of the datastore, manually creating an entity you can set a property of type "embedded entity" ... This is exactly what I am trying to do, but when adding a property to the property slice before saving (in the save method of the propertyloadsaver compatible struct) of type "datastore.Entity" with nil key and propertylist matching the slice of properties of the inner struct ... I received this "invalid value for a property with name cinfo" ... ...
Any idea ?