I am trying to take a cloudwatch_event and get it into a go struct. I have a CloudwatchEvent struct and inside of that is a blob of json that I need to get into another struct. The first level of the struct seems to work fine, but there is a parsing error when it tries to access the nested json.
This is my sample event. I am trying to get down to detail > EC2InstanceId I think I will also need the status code.
{
"version": "0",
"id": "3e3c153a-8339-4e30-8c35-687ebef853fe",
"detail-type": "EC2 Instance Launch Successful",
"source": "aws.autoscaling",
"account": "123456789012",
"time": "2015-11-11T21:31:47Z",
"region": "us-east-1",
"resources": [
"arn:aws:autoscaling:us-east-1:123456789012:autoScalingGroup:eb56d16b-bbf0-401d-b893-d5978ed4a025:autoScalingGroupName/sampleLuanchSucASG",
"arn:aws:ec2:us-east-1:123456789012:instance/i-b188560f"
],
"detail": {
"StatusCode": "InProgress",
"AutoScalingGroupName": "sampleLuanchSucASG",
"ActivityId": "9cabb81f-42de-417d-8aa7-ce16bf026590",
"Details": {
"Availability Zone": "us-east-1b",
"Subnet ID": "subnet-95bfcebe"
},
"RequestId": "9cabb81f-42de-417d-8aa7-ce16bf026590",
"EndTime": "2015-11-11T21:31:47.208Z",
"EC2InstanceId": "i-b188560f",
"StartTime": "2015-11-11T21:31:13.671Z",
"Cause": "At 2015-11-11T21:31:10Z a user request created an AutoScalingGroup changing the desired capacity from 0 to 1. At 2015-11-11T21:31:11Z an instance was started in response to a difference between desired and actual capacity, increasing the capacity from 0 to 1."
}
}
Since the aws-go-lambda library does not seem to handle these events i created two structs.
type CloudWatchEvent struct {
Version string `json:"version"`
ID string `json:"id"`
DetailType string `json:"detail-type"`
Source string `json:"source"`
AccountID string `json:"account"`
Time time.Time `json:"time"`
Region string `json:"region"`
Resources []string `json:"resources"`
Detail CloudWatchDetails `json:"detail"`
}
type CloudWatchDetails struct {
Detail struct {
StatusCode string `json:"StatusCode"`
AutoScalingGroupName string `json:"AutoScalingGroupName"`
ActivityID string `json:"ActivityId"`
Details struct {
AvailabilityZone string `json:"Availability Zone"`
SubnetID string `json:"Subnet ID"`
} `json:"Details"`
RequestID string `json:"RequestId"`
EndTime time.Time `json:"EndTime"`
EC2InstanceID string `json:"EC2InstanceId"`
StartTime time.Time `json:"StartTime"`
Cause time.Time `json:"Cause"`
} `json:"detail"`
}
I seem to be able to address event.Version or event.Id fine but when I try and address event.Detail.EC2InstanceId I get what looks like a byte object.
You were double nesting Detail property. Also "Cause" property in the JSON is a string and not a time.Time, you might want to change it.
That should do the trick.
type CloudWatchEvent struct {
Version string `json:"version"`
ID string `json:"id"`
DetailType string `json:"detail-type"`
Source string `json:"source"`
AccountID string `json:"account"`
Time time.Time `json:"time"`
Region string `json:"region"`
Resources []string `json:"resources"`
Detail CloudWatchDetails `json:"detail"`
}
type CloudWatchDetails struct {
StatusCode string `json:"StatusCode"`
AutoScalingGroupName string `json:"AutoScalingGroupName"`
ActivityID string `json:"ActivityId"`
Details struct {
AvailabilityZone string `json:"Availability Zone"`
SubnetID string `json:"Subnet ID"`
} `json:"Details"`
RequestID string `json:"RequestId"`
EndTime time.Time `json:"EndTime"`
EC2InstanceID string `json:"EC2InstanceId"`
StartTime time.Time `json:"StartTime"`
Cause string `json:"Cause"`
}
Code in the playground
Related
I am having following data format:
{"time":"2022-08-24T06:00:00Z","duration":0,"level":"OK","data":{"series":[{"name":"gnb_kpi","tags":{"ID":"1017","_field":"Success_rate%","cluster_id":"ec17-1017","swversion":"6.0"},"columns":["time","_value"],"values":[["2022-08-24T06:00:00Z","100"]]}]},"previousLevel":"CRITICAL","recoverable":true}
I want to remove the _time field from the columns array and similarily the timestamp from values array. The output I want is like this:
{"time":"2022-08-24T06:00:00Z","duration":0,"level":"OK","data":{"series":[{"name":"gnb_kpi","tags":{"ID":"1017","_field":"Success_rate%","cluster_id":"ec17-1017","swVersion":"6.0"},"columns":["_value"],"values":[["100"]]}]},"previousLevel":"CRITICAL","recoverable":true}
I used the online service JSON-to-Go to generate a data structure that corresponds to your input. It produced
type AutoGenerated struct {
Time time.Time `json:"time"`
Duration int `json:"duration"`
Level string `json:"level"`
Data struct {
Series []struct {
Name string `json:"name"`
Tags struct {
ID string `json:"ID"`
Field string `json:"_field"`
ClusterID string `json:"cluster_id"`
Swversion string `json:"swversion"`
} `json:"tags"`
Columns []string `json:"columns"`
Values [][]interface{} `json:"values"`
} `json:"series"`
} `json:"data"`
PreviousLevel string `json:"previousLevel"`
Recoverable bool `json:"recoverable"`
}
The algorithm is simple:
parse JSON into the generated structure
iterate over series
find the position of time field in columns
remove the corresponding data elements from values
https://go.dev/play/p/rjnvmdBXCE4
Output is (beautified)
{
"time": "2022-08-24T06:00:00Z",
"duration": 0,
"level": "OK",
"data": {
"series": [
{
"name": "gnb_kpi",
"tags": {
"ID": "1017",
"_field": "Success_rate%",
"cluster_id": "ec17-1017",
"swversion": "6.0"
},
"columns": [
"_value"
],
"values": [
[
"100"
]
]
}
]
},
"previousLevel": "CRITICAL",
"recoverable": true
}
As you see, no time
I’m trying to post some JSON. Using the JSON-to-Go tool I have this struct defined:
type IssueSetState struct {
ID string `json:"id"`
CustomFields []struct {
Value struct {
Name string `json:"name"`
} `json:"value"`
Name string `json:"name"`
Type string `json:"$type"`
} `json:"customFields"`
}
I’m trying to populate it with some data that I can then pass into the http library:
jsonValues := &IssueSetState{
ID: resultEntityId.ID,
CustomFields: []{
Value: {
Name: "Fixed",
},
Name: "State",
Type: "StateIssueCustomField",
},
}
jsonEncoded := new(bytes.Buffer)
json.NewEncoder(jsonEncoded).Encode(jsonValues)
I keep getting errors like:
./main.go:245:19: syntax error: unexpected {, expecting type
./main.go:246:9: syntax error: unexpected :, expecting comma or }
./main.go:249:8: syntax error: unexpected : at end of statement
./main.go:251:4: syntax error: unexpected comma after top level declaration
I’m sure the mistake I’m making is a simple one, but I’m new to Go.
One possible way is to define named structs for every anonymous struct you have.
type IssueSetState struct {
ID string `json:"id"`
CustomFields []CustomField `json:"customFields"`
}
type CustomField struct {
Value Value `json:"value"`
Name string `json:"name"`
Type string `json:"type"`
}
type Value struct {
Name string `json:"name"`
}
Now you can create it like this:
IssueSetState{
ID: resultEntityId.ID,
CustomFields: []CustomField{
{
Value: Value{
Name: "Fixed",
},
Name: "State",
Type: "StateIssueCustomField",
},
{
Value: Value{
Name: "Fixed",
},
Name: "State",
Type: "StateIssueCustomField",
},
},
}
So you're initializing the jsonValue badly.
You can fix it in 2 ways:
https://play.golang.org/p/LFO4tOLyG60
making structures flat
https://play.golang.org/p/TyFfaMf7XeF
by repeating the structure definition when declaring value
The first one should be easier and clearer.
I am working on unit test for a restAPI implementation in Golang.
I need to pass an array of object into url.
Here is an example of struct I have:
type version struct {
Name string `json:"name"`
Ver string `json:"ver"`
}
type event struct {
ID string `json:"id"`
Title string `json:"Title"`
Description string `json:"Description"`
Versions []version `json:"versions"`
}
The sample json input i tested in postman will be look like this one
{
"id": "101",
"title": "This is simple Golang title for testing!",
"Description":"Sample code for REST api implementation in Golang 2021!",
"versions": [
{
"name": "pingPong",
"ver": "10.2"
},
{
"name": "Ninja",
"ver": "10.24"
}
]
}
My question is that how can i pass an array of objects as URL parameters.
I expect to have something like below but not how to fill the ending part i highlighted by the ...
url?ID=20&Title=urlTitle&Description=UrlDescription&...
I don't know how you want the URL like, so I wrote it myself in a way that you can change it any way you want, And let me add that I don't know how many versions you have, so I wrote in such a way that no matter how many versions you have, it can handle it.
package main
import (
"fmt"
"strings"
"encoding/json"
)
var jsonData string =
`{
"id": "101",
"title": "This is simple Golang title for testing!",
"Description":"Sample code for REST api implementation in Golang 2021!",
"versions": [
{
"name": "pingPong",
"ver": "10.2"
},
{
"name": "Ninja",
"ver": "10.24"
}
]
}`
type (
Event struct {
Id string `json:"id"`
Title string `json:"title"`
Description string `json:"description"`
Versions []Version `json:"versions"`
}
Version struct {
Name string `json:"name"`
Ver string `json:"ver"`
}
)
func fillVersions(event *Event, baseUrl string) string {
var finalUrl string = baseUrl
for index, value := range event.Versions {
restUrl := fmt.Sprintf("Version%d=%s-%s", index + 1, value.Name, value.Ver)
finalUrl = fmt.Sprintf(
finalUrl + "%s" + "&",
restUrl,
)
}
return strings.TrimRight(finalUrl, "&")
}
func main() {
var event Event
json.Unmarshal([]byte(jsonData), &event)
baseUrl := fmt.Sprintf(
"https://test.com/test?Id=%s&Title=%s&Description=%s&",
event.Id,
event.Title,
event.Description,
)
finalUrl := fillVersions(&event, baseUrl)
fmt.Println(finalUrl)
}
The output of the program is as follows:
https://test.com/test?Id=101&Title=This is simple Golang title for testing!&Description=Sample code for REST api implementation in Golang 2021!&Version1=pingPong-10.2&Version2=Ninja-10.24
I would also like to say that the last & will be removed, If you don't want to do this, Remove the following line and write as follows (also remove the strings library from the import scope):
return strings.TrimRight(finalUrl, "&") // remove this
return finalUrl // add this
model.go:
type First struct {
ID int `json:"id" gorm:"column:id;primary_key"`
Status string `json:"status" gorm:"column:status"`
SecondID int `json:"second_id" gorm:"column:second_id"`
SecondData Second `json:"second_data" gorm:"foreignKey:SecondID;references:ID"`
}
type Second struct {
ID int `json:"id" gorm:"column:second_id;primary_key"`
Status string `json:"status" gorm:"column:status"`
Description string `json:"description" gorm:"column:description"`
}
var res []model.First
db.Raw("first.*, second.* FROM first LEFT JOIN second ON first.second_id = second.second_id")
db.Preload("SecondData").Find(&res).Error
Output:
{
"id": 1,
"status": "A",
"second_id": 1
"second_data": {
"id": 1
"status": "B",
"description": "blablabla"
}
}
I don't really know how db.Preload() works. Why i should use db.Preload() to get "SecondData" every time i need do nested struct ? Are it's possible only use db.Row() or db.Table().Joins().Where().Find(), i mean's without db.Preload()?
If you want SecondData loaded every time when the First struct is loaded without using Preload, you might consider using hooks.
It might look something like this:
func (f *First) AfterFind(tx *gorm.DB) error {
return tx.First(&f.SecondData, f.SecondID).Error
}
So, when you load the First data, the AfterFind hook should be triggered.
So I have an Struct that holds data that has a AddedByUser which links to my User Struct.
What I want to be able to do it remove the UserLevel from the AddedByUser
Now I want to be able to do it from this function only, so using the json:"-" is not an option. That would remove it from all json output. I only want to remove it form this one function.
I should also say that these are Gorm models and when I have been trying to remove the 10 option (UserLevels) it only removes the outer data set not the UserLevel from all of the data.
{
"ID": 1,
"CreatedAt": "2019-01-08T16:33:09.514711Z",
"UpdatedAt": "2019-01-08T16:33:09.514711Z",
"DeletedAt": null,
"UUID": "00000000-0000-0000-0000-000000000000",
"Title": "title000",
"Information": "info999",
"EventDate": "2006-01-02T15:04:05Z",
"AddedByUser": {
"ID": 2,
"CreatedAt": "2019-01-08T15:27:52.435397Z",
"UpdatedAt": "2019-01-08T15:27:52.435397Z",
"DeletedAt": null,
"UUID": "b019df80-a7e4-4397-814a-795e7e84b4ca",
"Firstname": "Me",
"Surname": "admin",
"Password": "....",
"Email": "admin#email.co.uk",
"UserLevel": {
"ID": 0,
"CreatedAt": "0001-01-01T00:00:00Z",
"UpdatedAt": "0001-01-01T00:00:00Z",
"DeletedAt": null,
"LevelTitle": "",
"UserLevel": null
},
So this is what I have tried,
data := []models.MyData{}
data = append(data[0:2])
I have about 14 results, with out the append it loads all the results but with this is only loads two results. The idea was to remove either UpdateAt or Title. As I am not sure if the gorm model information is all 0 or if the slice sees them as 0,1,2,3,4 etc.
I have also tried to range over the slice of models, while I can access each of the sections, I can not seem to find a simple method to remove data by name from a struct? Maps seem to have that but not structs which I am not sure why?
Thanks.
UPDATE
This is the model I am using:
//Model
type MyData struct {
gorm.Model
UUID uuid.UUID
Title string
Information string
EventDate time.Time
AddedByUser Users `gorm:"ForeignKey:added_by_user_fk"`
AddedByUserFK uint
}
//Users Model
type Users struct {
gorm.Model
UUID uuid.UUID
Firstname string
Surname string
Password string
Email string
UserLevel UserLevels `gorm:"ForeignKey:user_level_fk" json:",omitempty"`
UserLevelFK uint
}
As mentioned in the comments, you cannot remove fields from a struct value, because that would yield a value of a different type.
However, you can set fields to their zero value. Combined with the omitempty JSON tag, you can exclude fields from the JSON encoding. To make this work properly, you have to change the UserLevel field to a pointer type (otherwise you end up with empty objects in the JSON document).
Types shortened for brevity:
package main
import (
"encoding/json"
"fmt"
)
type MyData struct {
Title string
AddedByUser Users
}
type Users struct {
ID int
UserLevel *UserLevels `json:",omitempty"` // pointer type with omitempty
}
type UserLevels struct {
LevelTitle string
}
func main() {
var x MyData
x.Title = "foo"
x.AddedByUser.ID = 2
x.AddedByUser.UserLevel = &UserLevels{}
f(x)
b, _ := json.MarshalIndent(x, "", " ")
fmt.Println("main:\n" + string(b))
}
func f(x MyData) {
// "unset" UserLevel. Since we are receiving a copy of MyData, this is
// invisible to the caller.
x.AddedByUser.UserLevel = nil
b, _ := json.MarshalIndent(x, "", " ")
fmt.Println("f:\n" + string(b))
}
// Output:
// f:
// {
// "Title": "foo",
// "AddedByUser": {
// "ID": 2
// }
// }
// main:
// {
// "Title": "foo",
// "AddedByUser": {
// "ID": 2,
// "UserLevel": {
// "LevelTitle": ""
// }
// }
// }
Try it on the playground: https://play.golang.org/p/trUgnYamVOA
Alternatively, you can define new types that exclude the AddedByUser field. However, since this field isn't at the top level, this is a lot of work, and it's easy to forget to update those types when new fields are added to the original types.
If the field were at the top level, the compiler would do most of the work for you, because types that only differ in their field tags can be directly converted to one another:
type MyData struct {
ID int
Title string
}
func main() {
var x MyData
x.ID = 1
x.Title = "foo"
f(x)
}
func f(x MyData) {
type data struct { // same as MyData, except the field tags
ID int
Title string `json:"-"`
}
b, _ := json.MarshalIndent(data(x), "", " ")
fmt.Println("main:\n" + string(b))
}