New to golang & trying to make a script for making bulk upload to Elasticsearch server. My json data set is something like this...
{
product_displayname: "LG Stylus 2 Plus K535D (16 GB, Brown)",
product_price: "24000.00",
popularity: "0.00",
barcode: "",
exclusive_flag: "0",
product_id: "176982",
product_name: "Stylus 2 Plus K535D (Brown)",
brand_name: "LG",
brand_id: "1",
product_spec : {
display_spec: [{
spec_id: "103",
sdv: "24000",
snv: "24000.0000"
}, {
spec_id: "104",
sdv: "GSM",
snv: "0.0000"
}],
filter_spec: [{
spec_id: "103",
sdv: "24000",
snv: "24000.0000"
}, {
spec_id: "105",
sdv: "Touch Screen",
snv: "0.0000"
}]
}
}
Golang Structure I made(by refering google & other online info) for the above dataset is like this...
type Product struct {
product_displayname string `json:"product_displayname"`
product_price string `json:"product_price"`
popularity string `json:"popularity"`
barcode string `json:"barcode"`
exclusive_flag string `json:"exclusive_flag"`
product_id string `json:"product_id"`
product_name string `json:"product_name"`
brand_name string `json:"brand_name"`
brand_id string `json:"brand_id"`
product_spec
}
type product_spec struct {
display_spec []display_speclist
filter_spec []filter_speclist
}
type display_speclist struct {
spec_id string `json:"spec_id"`
sdv string `json:"sdv"`
snv string `json:"snv"`
}
type filter_speclist struct {
spec_id string `json:"spec_id"`
sdv string `json:"sdv"`
snv string `json:"snv"`
}
But, whenever I'm trying to use above Structure with sample data in my bulk upload script I'm getting following error
github.com/crazyheart/elastic-bulk-upload/main.go:70: syntax error: missing operand
github.com/crazyheart/elastic-bulk-upload/main.go:70: unknown escape sequence
github.com/crazyheart/elastic-bulk-upload/main.go:71: syntax error: non-declaration statement outside function body
I feel like I'm making some mistake in mapping that nested field display_spec & filter_spec in golang structure. But can't able to figure out what it is.
main.go
package main
import (
"fmt"
"golang.org/x/net/context"
"gopkg.in/olivere/elastic.v5"
"strconv"
)
type Product struct {
ProductDisplayname string `json:"product_displayname"`
ProductPrice string `json:"product_price"`
Popularity string `json:"popularity"`
Barcode string `json:"barcode"`
ExclusiveFlag string `json:"exclusive_flag"`
ProductID string `json:"product_id"`
ProductName string `json:"product_name"`
BrandName string `json:"brand_name"`
BrandID string `json:"brand_id"`
ProductSpec struct {
DisplaySpec []struct {
SpecID string `json:"spec_id"`
Sdv string `json:"sdv"`
Snv string `json:"snv"`
} `json:"display_spec"`
FilterSpec []struct {
SpecID string `json:"spec_id"`
Sdv string `json:"sdv"`
Snv string `json:"snv"`
} `json:"filter_spec"`
} `json:"product_spec"`
}
func main() {
// Create a context
ctx := context.Background()
client, err := elastic.NewClient()
if err != nil {
fmt.Println("%v", err)
}
// Bulk upload code
n := 0
for i := 0; i < 1000; i++ {
bulkRequest := client.Bulk()
for j := 0; j < 10000; j++ {
n++
product_data := Product{product_displayname:"LG Stylus 2 Plus K535D (16 GB, Brown)",product_price:"24000.00",popularity:"0.00",barcode:"",exclusive_flag:"0",product_id:"17698276",product_name:"Stylus 2 Plus K535D (Brown)",brand_name:"LG",brand_id:"1",product_spec:{display_spec:[{spec_id:"103",sdv:"24000",snv:"24000.0000"},{spec_id:"104",sdv:"GSM",snv:"0.0000"}],filter_spec:[{spec_id:"103",sdv:"24000",snv:"24000.0000"},{spec_id:"105",sdv:"Touch Screen",snv:"0.0000"}]} }
req := elastic.NewBulkIndexRequest().Index("shopfront").Type("products").Id(strconv.Itoa(n)).Doc(product_data)
bulkRequest = bulkRequest.Add(req)
}
bulkResponse, err := bulkRequest.Do(ctx)
if err != nil {
fmt.Println(err)
}
if bulkResponse != nil {
fmt.Println(bulkResponse)
}
fmt.Println(i)
}
}
Workflow
1.- Validate your json (the one you posted is invalid).
2.- Build a proper struct, you can help yourself using this nice tool.
For your case
The structs appears to be fine except you're not exporting the struct fields by capitalizing the initial letter (Thanks #ANisus).
This (shorted) seems more natural.
type Product struct {
ProductDisplayname string `json:"product_displayname"`
ProductSpec struct {
DisplaySpec []struct {
SpecID string `json:"spec_id"`
Sdv string `json:"sdv"`
Snv string `json:"snv"`
} `json:"display_spec"`
FilterSpec []struct {
SpecID string `json:"spec_id"`
Sdv string `json:"sdv"`
Snv string `json:"snv"`
} `json:"filter_spec"`
} `json:"product_spec"`
}
Related
I am trying to create the new API with Go and Fiber framework. I am using MongoDB.
Please refer to my below structure
type Product struct {
Name string `json:"name"`
Code string `json:"code"`
Description string `json:"description"`
Brand string `json:"brand"`
Variants []map[string]string `json:"variants"`
Categories []int `json:"categories"`
}
The Variants field will have another set array of objects. But the fields will be dynamic. So I can't define the structure. Now this one I solved with []map[string][string].
My task here I have to form the CSV file with the below format.
[{
Name: "Rock",
Code: "RRR"
Description: "This is rock"
Brand: "R"
...variants
},
{
Name: "Rock",
Code: "RRR"
Description: "This is rock"
Brand: "R"
...variants
},
{
Name: "Rambo",
Code: "RAM"
Description: "This is Rambo"
Brand: "RA"
...variants
},
{
Name: "Rambo",
Code: "RAM"
Description: "This is Rambo"
Brand: "RA"
...variants
}]
Only variants will change for every row. Others will be remaining same
I formed the other fields except for the Variants. I can't create the struct because the data will be dynamic.
I have a lot of confusion
How to create the struct for dynamic fields
How to spread the Variants at the root level
I am from javascript. So, we used the spread operator. Here I am confused about how to do that in Golang.
Help me out, guys
#mkopriva is right
However you can use reflection to dynamically construct a new struct if you like
follow is a example:
package main
import (
"encoding/json"
"fmt"
"reflect"
)
type DynamicMap map[string]interface{}
type Product struct {
Name string `json:"name"`
Code string `json:"code"`
Description string `json:"description"`
Brand string `json:"brand"`
Variants map[string]string `json:"variants"`
Categories []int `json:"categories"`
}
func (j Product) MarshalJSON() ([]byte, error) {
m := DynamicMap{}
t := reflect.TypeOf(j)
v := reflect.ValueOf(j)
for i := 0; i < t.NumField(); i++ {
if t.Field(i).Tag.Get("json") == "variants" {
dyn := v.Field(i).Interface().(map[string]string)
for key, val := range dyn {
m[key] = val
}
} else if t.Field(i).Type.Kind() == reflect.String {
m[t.Field(i).Tag.Get("json")] = v.Field(i).String()
} else {
m[t.Field(i).Tag.Get("json")] = v.Field(i)
}
}
return json.Marshal(m)
}
func (j Product) UnmarshalJSON(data []byte) error {
fmt.Println("Unmarshal...")
return json.Unmarshal(data, &j)
}
func main() {
p := Product{Name: "aa", Variants: map[string]string{"a": "a", "b": "b"}}
marshal, err := json.Marshal(p)
if err != nil {
fmt.Printf("%v\n", err)
}
fmt.Printf("%s\n", marshal)
}
{"a":"a","b":"b","brand":"","categories":{},"code":"","description":"","name":"aa"}
type _getData struct {
Title string `json:"title" form:"title"`
Date string `json:"date" form:"date"`
Pages []struct {
Order int `json:"order" form:"title"`
Description string `json:"description" form:"description"`
} `json:"pages" form:"pages"`
func CreateDiary(c echo.Context) error {
var getData _getData
c.Bind(&getData)
fmt.Print(getData)
...
Receive the following data through c.FormParams command, please tell me how to bind it to _getData struct,
map[address:[미국 캘리포니아 산타클라라 카운티 쿠퍼티노 ] date:[2021-10-05] location:[37.32779072192643 -122.01981157064436] map_id:[0] pages[0][description]:[123123] pages[0][order]:[0] pages[1][description]:[123123] pages[1][order]:[1] tags[0][id]:[12] tags[0][tag_name]:[sdf] title:[123123]]
I want to get the data of pages as an array, but I am getting []
You can use 3rd party lib.
import "github.com/monoculum/formam/v3"
type MyFormData struct {
Pages []struct {
Order int `formam:"order"`
Description string `formam:"description"`
} `formam:"pages"`
Tags []struct {
TagName string `formam:"tag_name"`
Id string `formam:"id"`
} `formam:"tags"`
Title string `formam:"title"`
}
func HttpHandler(c echo.Context) error {
myFormData := MyFormData{}
form, err := c.FormParams()
if err != nil {
return err
}
dec := formam.NewDecoder(&formam.DecoderOptions{TagName: "formam"})
dec.Decode(form, &myFormData)
return c.JSON(200, myFormData)
}
I am trying to convert an old version jsonline into a new one (with a different structure).
Now the old file has the following structure
{"user": "myname", "uuid": "1242315425", "data": {"Niveau1": ["AUTRE", "RC"], "Niveau2": ["RA06"], "Niveau3": ["RA06_01"]}}
however, Niveau2 and Niveau3 are not always present and the length of the lists is not always the same.
The new file has a more complicated structure
{"user": "myname", "uuid": "1242315425","annotation":{"classifications":{"Niveau1":{"labels":[{"value":"AUTRE"}, {"value":"RC"}]}, "Niveau2": {"labels": [{"value":"RA06"}], "Niveau3": {"labels": [{"value":"RA06_01"}]}}}}
What I have done so far is (after parsing the files in appropriate structures) the following function
func convert(oldAnnots []AnnotV1) (newAnnots []AnnotV2) {
for _, element := range oldAnnots {
var newAnnot AnnotV2
newAnnot.User = element.User
newAnnot.Uuid = element.Uuid
if element.Data.Niveau1 != nil {
for i, annot1 := range element.Data.Niveau1 {
newAnnot.Annotation.Classif.Niveau1.Labels[i].Value = annot1
}
}
if element.Data.Niveau2 != nil {
for j, annot2 := range element.Data.Niveau2 {
newAnnot.Annotation.Classif.Niveau2.Labels[j].Value = annot2
}
}
if element.Data.Niveau3 != nil {
for k, annot3 := range element.Data.Niveau3 {
newAnnot.Annotation.Classif.Niveau3.Labels[k].Value = annot3
}
}
newAnnots = append(newAnnots, newAnnot)
}
return
}
However, I got the error saying the index [0] is out of range for my slice.
panic: runtime error: index out of range [0] with length 0
Definitions of the two structures are the following
type AnnotV1 struct {
Uuid string `json:"uuid"`
Data struct {
Niveau1 []string `json:"Niveau1"`
Niveau2 []string `json:"Niveau2"`
Niveau3 []string `json:"Niveau3"`
} `json:"data"`
User string `json:"user"`
}
and
type AnnotV2 struct {
Uuid string `json:"uuid"`
Annotation struct {
Classif struct {
Niveau1 struct {
Labels []struct {
Value string `json:value`
} `json:"labels"`
}
Niveau2 struct {
Labels []struct {
Value string `json:value`
} `json:"labels"`
}
Niveau3 struct {
Labels []struct {
Value string `json:value`
} `json:"labels"`
}
} `json:"classifications"`
} `json:"annotation"`
User string `json:"user"`
}
type Label struct {
Value string `json:"value"`
}
type AnnotV2 struct {
Uuid string `json:"uuid"`
Annotation struct {
Classif struct {
Niveau1 struct {
Labels []Label `json:"labels"`
}
Niveau2 struct {
Labels []Label `json:"labels"`
}
Niveau3 struct {
Labels []Label `json:"labels"`
}
} `json:"classifications"`
} `json:"annotation"`
User string `json:"user"`
}
pre-allocate the slice
if element.Data.Niveau2 != nil {
newAnnot.Annotation.Classif.Niveau2.Labels = make([]Label, len(element.Data.Niveau2))
for j, annot2 := range element.Data.Niveau2 {
newAnnot.Annotation.Classif.Niveau2.Labels[j].Value = annot2
}
}
or use append
if element.Data.Niveau2 != nil {
for _, annot2 := range element.Data.Niveau2 {
newAnnot.Annotation.Classif.Niveau2.Labels = append(newAnnot.Annotation.Classif.Niveau2.Labels, Label{annot2})
}
}
I am trying to parse a nested json string
I did get it to work by using multiple structs, but I am wondering if I can parse the JSON without using an extra struct.
type Events struct {
Events []Event `json:"events"`
}
type Event struct {
Name string `json:"name"`
Url string `json:"url"`
Dates struct {
Start struct {
LocalDate string
LocalTime string
}
}
}
type Embed struct {
TM Events `json:"_embedded"`
}
func TMGetEventsByCategory(location string, category string) {
parameters := "city=" + location + "&classificationName=" + category + "&apikey=" + api_key
tmUrl := tmBaseUrl + parameters
resp, err := http.Get(tmUrl)
var embed Embed
var tm Event
if err != nil {
log.Printf("The HTTP request failed with error %s\n", err)
} else {
data, _ := ioutil.ReadAll(resp.Body)
err := json.Unmarshal(data, &embed)
json.Unmarshal(data, &tm)
}
}
JSON Data looks like this:
{
"_embedded": {
"events": [],
},
"OtherStuff": {
}
}
Is it possible to get rid of the Embed struct and read straight to the events part of the json string?
What you're looking for here is json.RawMessage. It can help delay parsing of certain values, and in you case map[string]json.RawMessage should represent the top-level object where you can selectively parse values. Here's a simplified example you can adjust to your case:
package main
import (
"encoding/json"
"fmt"
)
type Event struct {
Name string `json:"name"`
Url string `json:"url"`
}
func main() {
bb := []byte(`
{
"event": {"name": "joe", "url": "event://101"},
"otherstuff": 15.2,
"anotherstuff": 100
}`)
var m map[string]json.RawMessage
if err := json.Unmarshal(bb, &m); err != nil {
panic(err)
}
if eventRaw, ok := m["event"]; ok {
var event Event
if err := json.Unmarshal(eventRaw, &event); err != nil {
panic(err)
}
fmt.Println("Parsed Event:", event)
} else {
fmt.Println("Can't find 'event' key in JSON")
}
}
In your case look for the _embedded key and then Unmarshal its value to Events
yes of course
type Embed struct {
TM []struct {
Name string `json:"name"`
Url string `json:"url"`
Dates struct {
Start struct {
LocalDate string
LocalTime string
}
} // add tag here if you want
} `json:"_embedded"`
}
I have a collection of documents that were inserted into Mongo looking something like this:
type Stats struct {
UserStatus string `json:"userStatus" bson:"userStatus"`
... a bunch more fields
}
type User struct {
ID bson.ObjectId `json:"-" bson:"_id"`
LastName string `json:"lastName" bson:"lastName"`
FirstName string `json:"firstName" bson:"firstName"`
Role string `json:"role" bson:"role"`
Tags []string `json:"tags" bson:"tags"`
... (a bunch more fields)
Stats UserStats `json:"stats" bson:"stats"`
}
I want to query it to get a specific report, so I tried this:
func UserNameReport() {
... get mongo session, etc.
// create struct of just the data I want returned
type UserNames struct {
LastName string `json:"lastName" bson:"lastName"`
FirstName string `json:"firstName" bson:"firstName"`
... etc
UserStats Stats `json:"stats" bson:"stats"`
}
projection := bson.M{"lastName":1, "firstName":1, etc}
result := []UserNames{}
err := x.Find({query user collection}).Select(projection).All(&result)
...
}
This works - my question is, how can I include just ONE field from the 'Stats' struct? In other words,
I essentially want the "projection" to be this:
projection := bson.M{"lastName":1, ..., "stats.userStatus":1} <-- stats.userStatus doesn't work
...
err := x.Find({query user collection}).Select(projection).All(&result)
I get the entire "Stats" embedded struct in the results - how can I filter out just one field from the sub-document in and put it into the result set?
Thanks!
It works perfectly for me, with MongoDB 2.6.5
The following code:
package main
import (
"fmt"
"gopkg.in/mgo.v2"
"gopkg.in/mgo.v2/bson"
"log"
)
type Statistics struct {
Url string
Hits int
}
type Person struct {
Num int
Uuid string
Name string
Stats []Statistics
}
func main() {
// Connect to the database
session, err := mgo.Dial("localhost")
if err != nil {
panic(err)
}
defer session.Close()
// Remove people collection if any
c := session.DB("test").C("people")
c.DropCollection()
// Add some data
err = c.Insert(
&Person{1, "UUID1", "Joe", []Statistics{Statistics{"a", 1}, Statistics{"b", 2}}},
&Person{2, "UUID2", "Jane", []Statistics{Statistics{"c", 3}, Statistics{"d", 4}}},
&Person{3, "UUID3", "Didier", []Statistics{Statistics{"e", 5}, Statistics{"f", 6}}})
if err != nil {
log.Fatal(err)
}
result := []Person{}
err = c.Find(bson.M{"$or": []bson.M{bson.M{"uuid": "UUID3"}, bson.M{"name": "Joe"}}}).Select(bson.M{"num": 1, "name": 1, "stats.hits": 1}).All(&result)
if err != nil {
log.Fatal(err)
}
fmt.Println(result)
}
results in:
[{1 Joe [{ 1} { 2}]} {3 Didier [{ 5} { 6}]}]
... which is precisely what I would expect.
Maybe this will help others - essentially I was trying to take a document with an embedded document and return a result set like I would do in SQL with a select a.LastName + ', ' + a.FirstName as Name, b.OtherData and in essence have a different 'table' / 'document'.
So here is my current solution - love to get better ones (more performant?) though!
I created a new struct and I'm using the 'mapstructure' library
import "github.com/goinggo/mapstructure"
type Stats struct {
UserStatus string `json:"userStatus" bson:"userStatus"`
... a bunch more fields
}
type User struct {
ID bson.ObjectId `json:"-" bson:"_id"`
LastName string `json:"lastName" bson:"lastName"`
FirstName string `json:"firstName" bson:"firstName"`
Role string `json:"role" bson:"role"`
Tags []string `json:"tags" bson:"tags"`
... (a bunch more fields)
Stats UserStats `json:"stats" bson:"stats"`
}
type MyReportItem struct {
FirstName string `json:"firstName" jpath:"firstName"`
LastName string `json:"lastName" jpath:"lastName"`
Status string `json:"status" jpath:"stats.userStatus"`
}
type MyReport struct {
Results []MyReportItem `json:"results"`
}
func xxxx(w http.ResponseWriter, r *http.Request) {
var users MyReportItem
// the results will come back as a slice of map[string]interface{}
mgoResult := []map[string]interface{}{}
// execute the query
err := c.Find(finder).Select(projection).All(&mgoResult)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
user := MyReportItem{}
// iterate through the results and decode them into the MyReport struct
for _, x := range mgoResult {
docScript, _ := json.Marshal(x)
docMap := map[string]interface{}{}
json.Unmarshal(docScript, &docMap)
err := mapstructure.DecodePath(docMap, &user)
if err == nil {
users.Results = append(users.Results, user)
}
}
... send back the results ...
_ := json.NewEncoder(w).Encode(&users)
}
Now I get a slice of objects in the form:
results: [
{
firstName: "John",
lastName: "Doe",
status: "active"
}
...
]
Instead of:
{
firstName: "John",
lastName: "Doe",
stats: {
status: "active"
}
}