I've been using Protobuf for object definitions to be used when communicating with DynamoDB. Until now, such objects would look like this:
type Record struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Id string `protobuf:"bytes,2,opt,name=id,json=id,proto3" dynamodbav:"id,omitempty" json:"id,omitempty"`
Name string `protobuf:"bytes,3,opt,name=name,json=name,proto3" dynamodbav:"name,omitempty" json:"name,omitempty"`
OrgId string `protobuf:"bytes,4,opt,name=org_id,json=orgId,proto3" dynamodbav:"org_id,omitempty" json:"org_id,omitempty"`
AccountId string `protobuf:"bytes,7,opt,name=account_id,json=accountId,proto3" dynamodbav:"account_id,omitempty" json:"account_id,omitempty"`
Address string `protobuf:"bytes,9,opt,name=address,proto3" dynamodbav:"address,omitempty" json:"address,omitempty"`
BillingTitle string `protobuf:"bytes,10,opt,name=billing_title,json=billingTitle,proto3" dynamodbav:"billing_title,omitempty" json:"billing_title,omitempty"`
Language string `protobuf:"bytes,11,opt,name=language,proto3" dynamodbav:"language,omitempty" json:"language,omitempty"`
Personal string `protobuf:"bytes,12,opt,name=personal,proto3" dynamodbav:"personal,omitempty" json:"personal,omitempty"`
Phonenumber string `protobuf:"bytes,13,opt,name=phonenumber,proto3" dynamodbav:"phonenumber,omitempty" json:"phonenumber,omitempty"`
Postalcode string `protobuf:"bytes,14,opt,name=postalcode,proto3" dynamodbav:"postalcode,omitempty" json:"postalcode,omitempty"`
ProjectId string `protobuf:"bytes,15,opt,name=project_id,json=projectId,proto3" dynamodbav:"project_id,omitempty" json:"project_id,omitempty"`
Remarks string `protobuf:"bytes,16,opt,name=remarks,proto3" dynamodbav:"remarks,omitempty" json:"remarks,omitempty"`
}
The table I'm adding these objects to has a global secondary index with project_id as the sort-key but this field isn't always populated. So, when I went to do a PutObject operation on the table, like this:
record := Record {
Id: "A000",
Name: "Some Record",
OrgId: "O000"
}
attrs, err := dynamodbattribute.MarshalMap(&record)
if err != nil {
panic(err)
}
if _, err := conn.PutItem(&dynamodb.PutItemInput{
TableName: aws.String(tableName),
Item: attrs
}); err != nil {
panic(err)
}
this code wouldn't panic and I'd see the value in DynamoDB. However, after removing the omitempty clause from all the dynamodbav tags, I notice that this now fails with the following error:
ValidationException: Invalid attribute value type
status code: 400, request id: 6a626232-fcd4-4999-afe4-3df5769ce1b2
After some further investigation, I see that the object is serialized as:
map[account_id:{
NULL: true
} address:{
NULL: true,
} billing_title:{
NULL: true
} id:{
S: "A000"
} name:{
S: "Some Record"
} language:{
NULL: true
} org_id:{
S: "O000"
} personal:{
NULL: true
} phonenumber:{
NULL: true
} postalcode:{
NULL: true
} project_id:{
NULL: true
} remarks:{
NULL: true
}
when omitempty is not included but serializes to:
map[id:{
S: "A000"
} name:{
S: "Some Record"
} org_id:{
S: "O000"
}
when omitempty is included.
I'd like to get the former with default values rather than NULL: true but I don't see any serialization options for this. Is there a way to do this without implementing the DynamoDB marshaler interface?
After some investigation, I found the source code for the Marshal function, here. According to this, there's a field called NullEmptyString which I can set to tell the encoder that I want empty strings to be sent as strings, rather than as null attributes. However, if I set this I cannot use Marshal, MarshalMap and MarshalList directly. So, I copied them and wrote my own:
func MarshalMap(in interface{}) (map[string]*dynamodb.AttributeValue, error) {
av, err := getEncoder().Encode(in)
if err != nil || av == nil || av.M == nil {
return map[string]*dynamodb.AttributeValue{}, err
}
return av.M, nil
}
func getEncoder() *dynamodbattribute.Encoder {
encoder := dynamodbattribute.NewEncoder()
encoder.NullEmptyString = false
return encoder
}
Related
I'm trying to add string "Employee" to my existing JSON response. Also, we need to be able to generate this version of json based on an user condition. Only if the user condition is met, I need to generate second version of json with string "Employee" added. If not the first version without string "Employee" should be generated. How can I achieve it with out updating the existing struct and how can I check this with if clause to check for the condition and then generate json based on it?
Below is my existing json response in go
[
{
"EmpId":{
"String":"ABCD",
"Valid":true
},
"Department":{
"Float64":0,
"Valid":true
}
}
]
How can I get my json response like below with out changing existing struct based on input parameter?
{
"Employee":[
{
"EmpId":{
"String":"ABCD",
"Valid":true
},
"Department":{
"Float64":0,
"Valid":true
}
}
]
}
Below is my code:
Step 1: model folder
type EmployeeWithRoot struct {
Employee []Employee
}
type Employee struct {
EmpNbr sql.NullString `json:"EmpNbr"`
DateofJoin sql.NullString `json:"DateofJoin"`
DeptId sql.NullString `json:"DeptId"`
DeptName sql.NullString `json:"DeptName"`
}
Step 2: code folder
func GetEmpDetails(logRequestId string, logNestedLevel int, EmpNbr string, DateofJoin string) ([]model.EmployeeWithRoot, error) {
logFunctionFunctionName := "code.GetEmpDetails"
logStartTime := time.Now()
logNestedLevel++
defer configurations.TimeTrack(logFunctionFunctionName, logRequestId, logStartTime, logNestedLevel)
rows, err := db.Query(utils.SELECT_OF_EMP_AGGR, EmpNbr, DateofJoin, DeptId, DeptName)
if err != nil {
return nil, err
}
defer rows.Close()
var e []model.EmployeeWithRoot
for rows.Next() {
var x model.EmployeeWithRoot
err := rows.Scan(&x.Employee.EmpNbr, &x.Employee.DateofJoin, &x.Employee.DeptId,&x.Employee.DeptName)
if err != nil {
return nil, err
}
e = append(e, x)
}
err = rows.Err()
if err != nil {
return nil, err
}
return e, nil
}
STEP 3: API folder
Employee, err := code.GetEmpDetails(logRequestId, logNestedLevel, EmpNbr, DateofJoin)
if err != nil {
log.Panic(err)
}
marshalDataForRequestContentType(logRequestId, logNestedLevel, w, r, Employee)
I'm getting the below error.
x.Employee.EmpNbr undefined (type []model.Employee has no field or method EmpNbr)
x.Employee.DateofJoin undefined (type []model.Employee has no field or method DateofJoin)enter code here
x.Employee.DeptId undefined (type []model.Employee has no field or method DeptId)
x.Employee.DeptName undefined (type []model.Employee has no field or method DeptName)
Considering you're just wrapping it it an outer object, I don't see any reason you'd need to change the existing struct, just wrap it in a new one. I'll have to make some guesses/assumptions here since you've only shown the JSON and not the Go code that produces it, but assuming your existing JSON is produced by marshaling something like var response []Employee, the desired JSON could be produced in your condition by marshaling instead:
json.Marshal(struct{Employee []Employee}{response})
Working example: https://go.dev/play/p/vwDvxnQ96G_2
Use string concatenation:
func addRoot(json string) string {
return `{ "Employee":` + json + `}`
}
Run an example on the GoLang playground.
Here's the code if you are working with []byte instead of string:
func addRoot(json []byte) []byte {
const prefix = `{ "Employee":`
const suffix = `}`
result := make([]byte, 0, len(prefix)+len(json)+len(suffix))
return append(append(append(result, prefix...), json...), suffix...)
}
Run this example on the GoLang playground.
If you have some JSON in a byte slice ([]byte) then you can just add the outer element directly - e.g. (playground):
existingJSON := []byte(`[
{
"EmpId":{
"String":"ABCD",
"Valid":true
},
"Department":{
"Float64":0,
"Valid":true
}
}
]`)
b := bytes.NewBufferString(`{"Employee":`)
b.Write(existingJSON)
b.WriteString(`}`)
fmt.Println(string(b.Bytes()))
If this is not what you are looking for please add further details to your question (ideally your attempt as a minimal, reproducible, example)
I'm new to Go, so this might be very easy, but I can't find it. I have an entity Page with these two properties:
type Page struct {
Title string `form:"title" binding:"required"`
Active bool
}
Now, if you don't send a title we get a (not very pretty, but acceptable*):
Key: 'Page.Title' Error:Field validation for 'Title' failed on the 'required' tag.
Now, if I send this to the endpoint:
{
"title": "I'm a valid title",
"active": "I'm not a boolean at all!"
}
We get this:
json: cannot unmarshal string into Go struct field Page.Active of type bool
Which, IMO, is giving way too much info. What is the standard practice in Go to validate user input?
I was first making a page-validor.go with some checks, then I found this, but I'm not sure what is good practice in Go.
How do I validate this properly? Should I find check what is provided and then try to move it into the struct and validate the actual contents?
I am using GinGonic
* I've found a way to unwrap the errors and make it nicer
Write custom JSON Unmarshaller method for the type Page and inside UnmarshalJSON(bytes []byte) method, you can unmarshal the JSON bytes to map[string]interface{} and then validate the types you need with the JSON field keys.
An example of the JSON Unmarshaller looks like below.
type Page struct {
Title string `form:"title" binding:"required"`
Active bool
}
func (p *Page) UnmarshalJSON(bytes []byte) error {
var data map[string]interface{}
err := json.Unmarshal(bytes, &data)
if err != nil {
return err
}
actv, _ := data["active"]
if reflect.TypeOf(actv).Kind() != reflect.Bool {
return errors.New("active field should be a boolean")
}
p.Active = actv.(bool)
return nil
}
See the full example here in Playground.
After some more research, I've implemented Go-map-schema.
var page Page
src := make(map[string]interface{})
json.Unmarshal(jsonData, &src)
results, _ := schema.CompareMapToStruct(page, src, nil)
fmt.Println(results.MissingFields)
fmt.Println(results.MismatchedFields)
This works simple with the standard notations for an struct:
type Page struct {
Title string `json:"title" validator:"required"`
Active bool `json:"metaRobotsFollow" validate:"required,bool"`
}
You should use validator v10 available with Go
validator documentation go
For your use case you can use boolean validator
https://pkg.go.dev/github.com/go-playground/validator/v10#hdr-Boolean
type Page struct {
Title string `json:"title" binding:"required"`
Active bool `json:"active" binding:"required,boolean"`
}
Below is sample Test case with one positive and negative
func TestPage(t *testing.T) {
tests := []struct {
testName string
input string
wantErr bool
}{
{
testName: "positive test",
input: `{
"title": "first book title",
"active": false
}`,
wantErr: false,
}, {
testName: "wrong boolean",
input: `{
"title": "second book title",
"active": falsee
}`,
wantErr: false,
},
}
for _, tt := range tests {
t.Run(tt.input, func(t *testing.T) {
var p Page
b := []byte(tt.input)
err := json.Unmarshal(b, &p)
assert.Nil(t, err, "got error %v", err)
})
}
I'm relatively new to Go and am working on building out a request decoder. The request comes in JSON format and we decode that to a map[string]interface{}. We then pass that object data in to be decoded to our own ProcessRequest struct. As I said I'm new so I am reusing some logic in similar parts of the code wrote by previous developers. Essentially we are checking the map for the necessary pieces and then setting and returning those. Can someone explain to me why I am getting the titled error? Would I have to set the items all the way down to base structs that no longer have any nested? Is there a better way to accomplish what I want? Here is the code and the related structs. It is highlighting the error on the return of the model.ProcessRequest. TYIA
type ProcessRequest struct {
RequestID string
Message *message.Message
Rule *Rule
Options *ProcessOptions
//TODO: Context EvaluatorContext
//TODO: Links
}
type Message struct {
ID int
Key string
Source string
Headers *HeaderSet
Categories *CategorySet
Properties *PropertySet
Streams *StreamSet
}
type RuleAction struct {
Name string
Expression map[string]interface{}
}
type RuleLink struct {
LinkID int
Condition map[string]interface{}
TargetRuleID int
}
type Rule struct {
Name string
Code string
Actions []RuleAction
Links []RuleLink
}
type object = map[string]interface{}
func DecodeProcessRequest(dataObject map[string]interface{}) (*model.ProcessRequest, error) {
var (
requestID string
message *message.Message
rule *model.Rule
options *model.ProcessOptions
err error
)
if reqIDSrc, ok := dataObject["requestId"]; ok {
if requestID, err = converter.ToString(reqIDSrc); err != nil {
return nil, errors.Wrapf(err, "Error reading property 'requestID'")
}
if requestID == "" {
return nil, errors.Errorf("Property 'requestID' is an empty string")
}
} else {
return nil, errors.Errorf("Missing required property 'requestID'")
}
if messageSrc, ok := dataObject["message"]; ok {
messageData, ok := messageSrc.(object)
if !ok {
return nil, errors.Errorf("Error reading property 'message': Value is not an object")
}
if message, err = DecodeMessage(messageData); err != nil {
return nil, errors.Wrapf(err, "Error reading property 'message'")
}
} else {
return nil, errors.Errorf("Missing required property 'message'")
}
if ruleSrc, ok := dataObject["rule"]; ok {
ruleObj, ok := ruleSrc.(object)
if !ok {
return nil, errors.Errorf("Error reading property 'rule': Value is not an object")
}
if rule, err = DecodeRule(ruleObj); err != nil {
return nil, errors.Wrapf(err, "Error reading 'rule' during decoding")
}
} else {
return nil, errors.Errorf("Missing required property 'requestID'")
}
// Parse plain object to a Message struct
return &model.ProcessRequest{
requestID,
message,
rule,
options,
}, nil
}
super said in this comment:
In general, the warning says that you should prefer to use the syntax ProcessRequest{ RequestID: requestID, ... }. Naming the keys instead of unkeyed values.
That worked for me. Also the explanation by kostix in this comment really helped.
Basically the idea is that if you use "unkeyed" way of defining struct literals, the meaning of your definitions depends on the way the fields of the underlying type are layed out. Now consider that your type has three fields of type string in a certain order. Now a couple of iterations down the road some programmer moves the second field to the 1st position—your literal will still compile but will end up defining a completely different value at runtime.
I want to save a slice of structs in Google Cloud Datastore (Firestore in Datastore mode).
Take this Phonebook and Contact for example.
type Contact struct {
Key *datastore.Key `json:"id" datastore:"__key__"`
Email string `json:"email" datastore:",noindex"`
Name string `json:"name" datastore:",noindex"`
}
type Phonebook struct {
Contacts []Contact
Title string
}
Saving and loading this struct is no problem as the Datastore library takes care of it.
Due to the presence of some complex properties in my actual code, I need to implement PropertyLoadSaver methods.
Saving the Title property is straightforward. But I have problems storing the slice of Contact structs.
I tried using the SaveStruct method:
func (pb *Phonebook) Save() ([]datastore.Property, error) {
ps := []datastore.Property{
{
Name: "Title",
Value: pb.Title,
NoIndex: true,
},
}
ctt, err := datastore.SaveStruct(pb.Contacts)
if err != nil {
return nil, err
}
ps = append(ps, datastore.Property{
Name: "Contacts",
Value: ctt,
NoIndex: true,
})
return ps, nil
}
This code compiles but doesn't work.
The error message is datastore: invalid entity type
Making a slice of Property explicitly also does not work:
func (pb *Phonebook) Save() ([]datastore.Property, error) {
ps := []datastore.Property{
{
Name: "Title",
Value: pb.Title,
NoIndex: true,
},
}
cttProps := datastore.Property{
Name: "Contacts",
NoIndex: true,
}
if len(pb.Contacts) > 0 {
props := make([]interface{}, 0, len(pb.Contacts))
for _, contact := range pb.Contacts {
ctt, err := datastore.SaveStruct(contact)
if err != nil {
return nil, err
}
props = append(props, ctt)
}
cttProps.Value = props
}
ps = append(ps, cttProps)
return ps, nil
}
Making a slice of Entity does not work either:
func (pb *Phonebook) Save() ([]datastore.Property, error) {
ps := []datastore.Property{
{
Name: "Title",
Value: pb.Title,
NoIndex: true,
},
}
cttProps := datastore.Property{
Name: "Contacts",
NoIndex: true,
}
if len(pb.Contacts) > 0 {
values := make([]datastore.Entity, len(pb.Contacts))
props := make([]interface{}, 0, len(pb.Contacts))
for _, contact := range pb.Contacts {
ctt, err := datastore.SaveStruct(contact)
if err != nil {
return nil, err
}
values = append(values, datastore.Entity{
Properties: ctt,
})
}
for _, v := range values {
props = append(props, v)
}
cttProps.Value = props
}
ps = append(ps, cttProps)
return ps, nil
}
Both yielded the same error datastore: invalid entity type
Finally I resorted to using JSON. The slice of Contact is converted into a JSON array.
func (pb *Phonebook) Save() ([]datastore.Property, error) {
ps := []datastore.Property{
{
Name: "Title",
Value: pb.Title,
NoIndex: true,
},
}
var values []byte
if len(pb.Contacts) > 0 {
js, err := json.Marshal(pb.Contacts)
if err != nil {
return nil, err
}
values = js
}
ps = append(ps, datastore.Property{
Name: "Contacts",
Value: values,
NoIndex: true,
})
return ps, nil
}
Isn't there a better way of doing this other than using JSON?
I found this document and it mentions src must be a struct pointer.
The only reason you seem to customize the saving of PhoneBook seems to be to avoid saving the Contacts slice if there are no contacts. If so, you can just define your PhoneBook as follows and directly use SaveStruct on the PhoneBook object.
type Phonebook struct {
Contacts []Contact `datastore:"Contacts,noindex,omitempty"`
Title string `datastore:"Title,noindex"`
}
I'm new to Go and I'm facing issues in accessing a struct with a variable
I have this function decodeUser. My task is to check whether the keys are present in the request. So this is what I did. I've added a comment where I got the error.
func decodeUser(r *http.Request) (root.User, []string, error) {
var u root.User
if r.Body == nil {
return u, []string{}, errors.New("no request body")
}
decoder := json.NewDecoder(r.Body)
checks := []string{
"UserName",
"FirstName",
"LastName",
"Email",
}
emptyFields := []string{}
for _, check := range checks {
// i'm having problem over here `u[check]` it's showing (invalid
operation: u[check] (type root.User does not support
indexing))
if u[check] == nil {
emptyFields = append(emptyFields, check)
}
}
err := decoder.Decode(&u)
return u, emptyFields, err
}
Just in case I added root.User here's structure for it
type User struct {
ID string
Username string
Password string
FirstName string
LastName string
Email string
PhoneNumber string
PhoneNumberExtension string
DOB time.Time
AboutMe string
}
The problem occurs as it doesn't allow me to access struct by a variable and I can't use this method which is u.check. So basically how should I make u[check] work?
I would suggest you manually check for zero values since it seems that you already know the fields that needs to be non-zero at compile time. However, if that is not the case, here is a simple function (using reflection) that will check for zero values in a struct.
func zeroFields(v interface{}, fields ...string) []string {
val := reflect.Indirect(reflect.ValueOf(v))
if val.Kind() != reflect.Struct {
return nil
}
var zeroes []string
for _, name := range fields {
field := val.FieldByName(name)
if !field.IsValid() {
continue
}
zero := reflect.Zero(field.Type())
// check for zero value
if reflect.DeepEqual(zero.Interface(), field.Interface()) {
zeroes = append(zeroes, name)
}
}
return zeroes
}
func main() {
x := User{
Email: "not#nil",
}
fmt.Println(zeroFields(&x, "ID", "Username", "Email", "Something", "DOB"))
}
Which would output:
[ID Username DOB]
Playground
This is what worked for me
for _, check := range checks {
temp := reflect.Indirect(reflect.ValueOf(&u))
fieldValue := temp.FieldByName(string(check))
if (fieldValue.Type().String() == "string" && fieldValue.Len() == 0) || (fieldValue.Type().String() != "string" && fieldValue.IsNil()) {
fmt.Println("EMPTY->", check)
emptyFields = append(emptyFields, check)
}
}