Is it possible in Go to create a struct type programmatically (i.e. not in the compiled source code)?
We have a particular use case where a type will be created via user-defined metadata (so the schema/types are not known in advance)
and will vary for every customer. We would then need to auto-generate REST services for those and persist them in a NoSQL backend.
We would also need to define different dynamic validators per field (e.g. mandatory, regex, max/min size, max/min value, a reference to another type instance, etc.)
I was wondering if something similar is possible in the Go?
Edit 1:
For example
From frontend in JSON
For customer 1:
{
"id":1,
"patientid":1,
"name":"test1",
"height":"160",
"weight":"70",
"temp":"98"
}
For customer 2:
{
"id":2,
"patientid":1,
"name":"test1",
"height":"160",
"weight":"70"
}
For customer 3
may be different new fields will add
Backend
// For One customer need to have these fields
type Vitalsigns struct {
ID int64 `datastore:"-"`
PatientID int64 `json:"patientid,omitempty"`
Name string `json:"name,omitempty"`
Height string `json:"height,omitempty"`
Weight string `json:"weight,omitempty"`
Temp string `json:"temp,omitempty"`
}
// Another need to have these fields
type Vitalsigns struct {
ID int64 `datastore:"-"`
PatientID int64 `json:"patientid,omitempty"`
Name string `json:"name,omitempty"`
Height string `json:"height,omitempty"`
Weight string `json:"weight,omitempty"`
}
//CreateVitalsignsHandler is to create vitals for a patient
func CreateVitalsignsHandler(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
//Creating the Vitalsigns
kinVitalsigns := &Vitalsigns{}
ctx := appengine.NewContext(r)
if err := json.NewDecoder(r.Body).Decode(kinVitalsigns); err != nil {
RespondErr(w, r, http.StatusInternalServerError, err.Error())
return
}
//Getting namespace
namespace := ps.ByName("namespace")
ctx, err := appengine.Namespace(ctx, namespace)
if err != nil {
log.Infof(ctx, "Namespace error from CreateVitalsignsHandler")
RespondErr(w, r, http.StatusInternalServerError, err.Error())
return
}
//Geting the patientID
pID, err := strconv.Atoi(ps.ByName("id"))
if err != nil {
log.Infof(ctx, "Srting to Int64 conversion error from CreateVitalsignsHandler")
RespondErr(w, r, http.StatusInternalServerError, err.Error())
return
}
patientID := int64(pID)
kinVitalsigns.PatientID = patientID
//Generating the key
vitalsignsKey := datastore.NewIncompleteKey(ctx, "Vitalsigns", nil)
//Inserting the data to the datastore
vk, err := datastore.Put(ctx, vitalsignsKey, kinVitalsigns)
if err != nil {
log.Infof(ctx, "Entity creation was failed from CreateVitalsignsHandler")
RespondErr(w, r, http.StatusInternalServerError, err.Error())
return
}
kinVitalsigns.ID = vk.IntID()
message := "Vitalsigns created successfully!! "
Respond(w, r, http.StatusOK, SuccessResponse{kinVitalsigns.ID, 0, "", message})
return
}
Edit: Your edit reveals you want to handle dynamic objects to put / retrieve from Google Datastore. For this it is completely unnecessary to create struct types at runtime, you may just use a dynamic map presented in this answer: How can I have dynamic properties in go on the google app engine datastore.
Original answer follows.
Note that if the types are known at compile time, best / most efficient is to create the types and compile them, so everything will be "static". You may create the types manually, or you may use go generate to automate the process.
Also note that you may not necessarily need struct types to model dynamic objects, many times maps may be sufficient.
If types are not known at compile time, and struct types are a must, read on.
Yes, it's possible to create "dynamic" struct types at runtime using Go's reflection, specifically with the reflect.StructOf() function.
Let's see a simple example, creating a struct type at runtime that has a Name string and an Age int field:
t := reflect.StructOf([]reflect.StructField{
{
Name: "Name",
Type: reflect.TypeOf(""), // string
},
{
Name: "Age",
Type: reflect.TypeOf(0), // int
},
})
fmt.Println(t)
v := reflect.New(t)
fmt.Printf("%+v\n", v)
v.Elem().FieldByName("Name").Set(reflect.ValueOf("Bob"))
v.Elem().FieldByName("Age").Set(reflect.ValueOf(12))
fmt.Printf("%+v\n", v)
This outputs (try it on the Go Playground):
struct { Name string; Age int }
&{Name: Age:0}
&{Name:Bob Age:12}
If you want to define validation rules, you may use a 3rd party lib for that, for example github.com/go-validator/validator. This package uses struct tags to specify validation rules, struct tags which you may also specify using reflection.
For example, if you want to specify that the Name must be at least 3 characters and 40 at most, and it may only contain letters of the English alphabet, and valid range for Age is 6..100 (both inclusive), this is how it would look like:
t := reflect.StructOf([]reflect.StructField{
{
Name: "Name",
Type: reflect.TypeOf(""), // string
Tag: reflect.StructTag(`validate:"min=3,max=40,regexp=^[a-zA-Z]*$"`),
},
{
Name: "Age",
Type: reflect.TypeOf(0), // int
Tag: reflect.StructTag(`validate:"min=6,max=100"`),
},
})
Printing this type would output (wrapped by me) (try it on the Go Playground):
struct { Name string "validate:\"min=3,max=40,regexp=^[a-zA-Z]*$\"";
Age int "validate:\"min=6,max=100\"" }
Once you create an instance of this struct, you can validate it using the validator.Validate() function, e.g.:
v := reflect.New(t)
v.Elem().FieldByName("Name").Set(reflect.ValueOf("Bob"))
v.Elem().FieldByName("Age").Set(reflect.ValueOf(12))
if errs := validator.Validate(v.Elem().Interface()); errs != nil {
// values not valid, deal with errors here
}
Related
Let's say I have the first struct as
type Person struct {
Name string `json:"person_name"`
Age int `json:"person_age"`
Data map[string]interface{} `json:"data"`
}
and I am trying to marshal an array of the above struct
Things work well till here and a sample response I receive is
[
{
"person_name":"name",
"person_age":12,"data":{}
},
{
"person_name":"name2",
"person_age":12,"data":{}
}
]
Now, I need to append another struct over here and the final response should like
[
{
"person_name":"name",
"person_age":12,"data":{}
},
{
"person_name":"name2",
"person_age":12,"data":{}
},
{
"newData":"value"
}
]
So can someone help on this and how i can achieve this ?
I tried by creating an []interface{} and then iterating over person to append each data, but the issue in this approach is that it makes the Data as null if in case it's an empty string.
I would need it be an empty map only.
Let me prefix this by saying this looks to me very much like you might be dealing with an X-Y problem. I can't really think of many valid use-cases where one would end up with a defined data-type that has to somehow be marshalled alongside a completely different, potentially arbitrary/freeform data structure. It's possible, though, and this is how you could do it:
So you just want to append a completely different struct to the data-set, then marshal it and return the result as JSON? You'll need to create a new slice for that:
personData := []Person{} // person 1 and 2 here
more := map[string]string{ // or some other struct
"newdata": "value",
}
allData := make([]any, 0, len(personData) + 1) // the +1 is for the more, set cap to however many objects you need to marshal
for _, p := range personData {
allData = append(allData, p) // copy over to this slice, because []Person is not compatible with []any
}
allData = append(allData, more)
bJSON, err := json.Marshal(allData)
if err != nil {
// handle
}
fmt.Println(string(bJSON))
Essentially, because you're trying to marshal a slice containing multiple different types, you have to add all objects to a slice of type any (short for interface{}) before marshalling it all in one go
Cleaner approaches
There are much, much cleaner approaches that allow you to unmarshal the data, too, assuming the different data-types involved are known beforehand. Consider using a wrapper type like so:
type Person struct {} // the one you have
type NewData {
NewData string `json:"newdata"`
}
type MixedData struct {
*Person
*NewData
}
In this MixedData type, both Person and NewData are embedded, so MixedData will essentially act as a merged version of all embedded types (fields with the same name should be overridden at this level). With this type, you can marshal and unmarshal the JSON accordingly:
allData := []MixedData{
{
Person: &person1,
},
{
Person: &person2,
},
{
NewData: &newData,
},
}
Similarly, when you have a JSON []byte input, you can unmarshal it same as you would any other type:
data := []MixedData{}
if err := json.Unmarshal(&data, in); err != nil {
// handle
}
fmt.Printf("%#v\n", data) // it'll be there
It pays to add some functions/getters to the MixedData type, though:
func (m MixedData) IsPerson() bool { return m.Person != nil }
func (m MixedData) Person() *Person {
if m.Person == nil {
return nil
}
cpy := *m.Person // create a copy to avoid shared pointers
return &cpy // return pointer to the copy
}
Do the same for all embedded types and this works like a charm.
As mentioned before, should your embedded types contain fields with the same name, then you should override them in the MixedData type. Say you have a Person and Address type, and both have an ID field:
type MixedData struct {
ID string `json:"id"`
*Person
*Address
}
This will set the ID value on the MixedData type, and all other (non-shared) fields on the corresponding embedded struct. You can then use the getters to set the ID where needed, or use a custom unmarshaller, but I'll leave that to you to implement
So I have this table Schema that looks like the following
CREATE TABLE artists (
id SERIAL PRIMARY KEY,
foo_user_id TEXT NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL,
time_span TIME_SPAN NOT NULL,
items artist [] NOT NULL
);
In addition, I also have the custom type created in PostgreSQL that is defined as follows
CREATE TYPE artist AS (
artist_name TEXT,
foo_artist_id TEXT,
artist_image_url TEXT,
artist_rank int
);
I am trying to query all rows that have the "foo_user_id" equal to what I pass into the function. Here is the sample code.
func GetHistoricalTopArtists(foo_user_id string) ([]TopArtists, error) {
// connect to DB etc..
// create prepared statement
stmtStr := `SELECT * FROM artists WHERE foo_user_id=$1`
// error check...
// iterate through all rows to get an array of []TopArtists
defer rows.Close()
for rows.Next() {
topArtist := new(TopArtists)
err := rows.Scan(&topArtist.Id, &topArtist.FooUserId, &topArtist.CreatedAt, &topArtist.TimeSpan, &topArtist.Artists)
if err != nil {
log.Fatalf("Something went wrong %v", err)
}
topArtists = append(topArtists, *topArtist)
}
}
To represent this data in Go I created the following structs
// Represents a row
type TopArtists struct {
Id int64 `json:"id" db:"id"`
FooUserId string `json:"foo_user_id" db:"foo_user_id"`
CreatedAt string `json:"created_at" db:"created_at"`
TimeSpan string `json:"time_span" db:"time_span"`
Artists []Artist `json:"items" db:"items"`
}
// Represents the artist column
type Artist struct {
ArtistName string `json:"artist_name"`
ArtistId string `json:"foo_artist_id"`
ArtistImageURL string `json:"artist_image_url"`
ArtistRank int `json:"artist_rank"`
}
When I call the function that does the query (the one I described above). I get the following error.
Scan error on column index 4, name "items": unsupported Scan, storing driver.Value type []uint8 into type *[]database.Artist.
I have a Value() function, but I am unsure how to implement a Scan() function for the array of the custom struct I have made.
Here is my Value() function, I have attempted to read documentation and similar posts on scanning arrays of primitive types (strings, int, etc) but I could not apply the logic to custom PostgreSQL types.
func (a Artist) Value() (driver.Value, error) {
s := fmt.Sprintf("(%s, %s, %s, %d)",
a.ArtistName,
a.FooArtistId,
a.ArtistImageURL,
a.ArtistRank)
return []byte(s), nil
}
#mkopriva - ...You need to declare a slice type, e.g. type ArtistSlice []Artist,
use that as the field's type, and implement the Value/Scan methods on
that.
Created custom Composite Types Artist in Postgresq has a strict struct as
{(david,38,url,1),(david2,2,"url 2",2)}
then you have to implement Value/Scan method with custom marshal/unmarshal algorithm
For example
type Artists []Artist
func (a *Artists) Scan(value interface{}) error {
source, ok := value.(string) // input example 👉🏻 {"(david,38,url,1)","(david2,2,\"url 2\",2)"}
if !ok {
return errors.New("incompatible type")
}
var res Artists
artists := strings.Split(source, "\",\"")
for _, artist := range artists {
for _, old := range []string{"\\\"","\"","{", "}","(",")"} {
artist = strings.ReplaceAll(artist, old, "")
}
artistRawData := strings.Split(artist, ",")
i, err := strconv.Atoi(artistRawData[1])
if err != nil {
return fmt.Errorf("parce ArtistRank raw data (%s) in %d iteration error: %v", artist, i, err)
}
res = append(res, Artist{
ArtistName: artistRawData[0],
ArtistId: artistRawData[1],
ArtistImageURL: artistRawData[2],
ArtistRank: i,
})
}
*a = res
return nil
}
I've defined a data structure like so:
type Person struct {
Name string `firestore:"name,omitempty"`
}
When I query all the documents in a collection I'd like to be able to attach the ID to the documents for later reference, but not necessarily have ID as an attribute stored in Firestore (unless its the only way). In javascript or python this is straightforward as the data structures are dynamic and I can just query the ID post get() and add it as a dynamic key/value. myObj.id = doc.id
How would I do this with Go?
package main
import (
"fmt"
"cloud.google.com/go/firestore"
"context"
"google.golang.org/api/iterator"
"log"
)
type Person struct {
Name string `firestore:"name,omitempty"`
}
func main() {
ctx := context.Background()
c, err := firestore.NewClient(ctx, "my-project")
if err != nil {
log.Fatalf("error: %v", err)
}
var people []Person
iter := c.Collection("people").Documents(ctx)
for {
doc, err := iter.Next()
if err == iterator.Done {
break
}
if err != nil {
log.Fatalf("error: %v", err)
}
var p Person
err = doc.DataTo(p)
if err != nil {
log.Fatalf("error: %v", err)
}
// id := doc.Ref.ID
people = append(people, p)
}
fmt.Println(people)
}
output, no ID
>> [{John Smith}]
I believe that the firestore struct tags works the same way as the tags in the encoding/json package. So a tag with a value of "-" would mean ignore the field.
So
type Person struct {
ID int `firestore:"-"`
Name string `firestore:"name,omitempty"`
}
should do the trick.
You can set ID yourself, but the firestore pkg will ignore it when reading/writing data.
If you want to store the firestore document ID on the Person type, your struct must have a declared field for it.
Golang firestore docs don't mention this explicitly, but since a firestore doc ID is not part of the document fields, the func (*DocumentSnapshot) DataTo does not populate the ID. Instead, you may get the document ID from the DocumentRef type and add it to Person yourself.
The doc also states that:
Note that this client supports struct tags beginning with "firestore:" that work like the tags of the encoding/json package, letting you rename fields, ignore them, or omit their values when empty
Therefore, if you want to omit the ID when marshaling back for firestore, your could use the tag firestore:"-"
The Person would look like this:
type Person struct {
ID string `firestore:"-"`
Name string `firestore:"name,omitempty"`
}
inside the loop:
var p Person
err := docSnap.DataTo(&p)
if err != nil {
// handle it
}
p.ID = doc.Ref.ID
I know there is struct in Go, but for all I know, you have to define struct
type Circle struct{
x,y,r float64
}
I am wondering how you can declare a new variable that doesn't exist in the struct
circle := new(Circle)
circle.color = "black"
You will need to use a map (of type map[string]interface{}) to work with dynamic JSON. Here is an example of creating a new map:
// Initial declaration
m := map[string]interface{}{
"key": "value",
}
// Dynamically add a sub-map
m["sub"] = map[string]interface{}{
"deepKey": "deepValue",
}
Unmarshalling JSON into a map looks like:
var f interface{}
err := json.Unmarshal(b, &f)
The code above would leave you with a map in f, with a structure resembling:
f = map[string]interface{}{
"Name": "Wednesday",
"Age": 6,
"Parents": []interface{}{
"Gomez",
"Morticia",
},
}
You will need to use a type assertion to access it, otherwise Go won't know it's a map:
m := f.(map[string]interface{})
You will also need to use assertions or type switches on each item you pull out of the map. Dealing with unstructured JSON is a hassle.
More information:
https://blog.golang.org/json-and-go
https://godoc.org/encoding/json#Unmarshal
I've started to work on this small repository https://github.com/Ompluscator/dynamic-struct
It's possible at this point to extend existing struct in runtime, by passing a instance of struct and modifying fields (adding, removing, changing types and tags).
Still in progress, so don't expect something huge :)
EDIT: At this point, work on library is done, and it looks stable for last a couple of months :)
You can do it using reflect package, check StructOf method it allows you to create a new struct from []reflect.StructField. Example:
func main() {
typ := reflect.StructOf([]reflect.StructField{
{
Name: "Height",
Type: reflect.TypeOf(float64(0)),
Tag: `json:"height"`,
},
{
Name: "Age",
Type: reflect.TypeOf(int(0)),
Tag: `json:"age"`,
},
})
v := reflect.New(typ).Elem()
v.Field(0).SetFloat(0.4)
v.Field(1).SetInt(2)
s := v.Addr().Interface()
w := new(bytes.Buffer)
if err := json.NewEncoder(w).Encode(s); err != nil {
panic(err)
}
fmt.Printf("value: %+v\n", s)
fmt.Printf("json: %s", w.Bytes())
r := bytes.NewReader([]byte(`{"height":1.5,"age":10}`))
if err := json.NewDecoder(r).Decode(s); err != nil {
panic(err)
}
fmt.Printf("value: %+v\n", s)
}
You can't. Go is statically typed, and does not allow such constructs.
Structs have a layout in memory that directly related to the definition, and there's no where to store such additional fields.
You can use a map instead. Moreover, you can use &circle as a key or part of a key, to associate map elements with arbitrary structs.
type key struct {
target interface{}
field string
}
x := make(map[key]string)
x[key{ target: circle, field: "color" }] = "black"
Here are my array of objects,
type PeopleCount []struct{
Name string
Count int
}
type Consultation []struct{
Name string
Opd_count int
Opinion_count int
Req_count int
}
How should I pass both the objects to html template and arrange them in table?
Define an anonymous struct with fields for people count and consultations and pass the struct to the template Execute method:
var data = struct {
PeopleCounts []PeopleCount
Consultations []Consultation
}{
PeopleCounts: p,
Consultations: c,
}
err := t.Execute(w, &data)
if err != nil {
// handle error
}
Use these fields in the template:
{{range .PeopleCounts}}{{.Name}}
{{end}}
{{range .Consultations}}{{.Name}}
{{end}}
Playground example
You can declare a named type for the template data. The advantage of the anonymous type declaration is that knowledge of the template data is localized to the function that invokes the template.
You can also use a map instead of a type:
err := t.Execute(w, map[string]interface{}{"PeopleCounts": p, "Consultations": c})
if err != nil {
// handle error
}
The disadvantage of using a map is that a typo in the template may not result in an error. For example, ``{{range .PopleConts}}{{end}}` silent does nothing.
The code above assumes that PeopleCount and Consultation are struct types and not slices of anonymous struct types:
type PeopleCount struct {
Name string
Count int
}
type Consultation struct {
Name string
Opd_count int
Opinion_count int
Req_count int
}
It's usually more convenient to give the element a named type than to give the slice a named type.
If you prefer, define an unexported struct with fields for people count and consultations and pass the struct to the template Execute method:
type viewModel struct {
PeopleCounts []PeopleCount
Consultations []Consultation
}
// ...
var data = viewModel{
PeopleCounts: p,
Consultations: c,
}
err := t.Execute(w, &data)
if err != nil {
// handle error
}
This approach is broadly similar to #Bravada's answer. It's merely a matter of personal taste whether to use a view-model type explicitly or anonymously.