Is it possible to pickle instances of structs in Golang - go

I am doing some machine learning in Golang. I am now hitting a wall, my trained classifier takes almost half a minute to train and want to save that instance of the classifier so that I do not have to train in from scratch every time. How should someone go about doing this is Golang?
FYI my classifier is a struct
When I do this type of stuff with python, it's very easy with pickle. Is there an equivalent?

Try gob or encoding/json to marshal your objects. After that, you can store the string to a file.
Here is an example to use json:
package main
import (
"encoding/json"
"fmt"
"os"
)
type Book struct {
Title string
Pages []*Page
}
type Page struct {
pageNumber int // remember to Capitalize the fields you want to marshal
Content string
}
func main() {
// First make a book with nested array of struct pointers
myBook := &Book{Title: "this is a title", Pages: make([]*Page, 0)}
for i := 0; i < 3; i++ {
myBook.Pages = append(myBook.Pages, &Page{i + 1, "words"})
}
// Open a file and dump JSON to it!
f1, err := os.Create("/tmp/file1")
enc := json.NewEncoder(f1)
err = enc.Encode(myBook)
if err != nil {
panic(err)
}
f1.Close()
// Open the file and load the object back!
f2, err := os.Open("/tmp/file1")
dec := json.NewDecoder(f2)
var v Book
err = dec.Decode(&v)
if err != nil {
panic(err)
}
f2.Close()
// Check
fmt.Println(v.Title) // Output: <this is a title>
fmt.Println(v.Pages[1].Content) // Output: <words>
// pageNumber is not capitalized so it was not marshaled
fmt.Println(v.Pages[1].pageNumber) // Output: <0>
}

Related

copy buffer into type struct based on data

My requirement is to convert byte stream into my structure type, which has predefined data lengths.
Below sample, I could convert byte into "Test" object and read data(func buffertoStruct demonstrates that),
but the problem is
I need to cast into data types based on the string or int lengths. which is not happening now.
I have lots of different structures like "type Test" and each struct will have a good amount of data variables. So copying bytes into struct Test variables based on sizes one by one doesn't work.
WHAT I am thinking about solution:
I am thinking of keeping all struct variables names, sizes, type in a data structure, and slice byte stream in loop based on that
type datadetails struct {
name string //ex: uniqnum, guid etc
size int //9 ,32
datatype string //int, string, etc int for uniqnum, string for guid
}
Question: Need a better solution than that?
here is my sample, so far.
package main
import (
"bytes"
"encoding/gob"
"log"
"fmt"
)
type Test struct{
uniqnum int64 //9 char in byte stream ex: 708008123 (int)
guid string //exactly is 32 -chars
flag byte //string 1 char
printnum string //6 char
}
func convertintorec(data []byte) {
fmt.Println(data)
//convert byte data into Test record
}
func buffertoStruct(gbuffer []byte,pobj *Test ) {
tmp :=bytes.NewBuffer(gbuffer)
dec := gob.NewDecoder(tmp)
err := dec.Decode(pobj)
if err != nil {
log.Fatal("decode error:", err)
}
}
func main() {
bdata := []byte("9885455612guidstartshere12345678901234567F")
convertintorec(bdata)
}
This should get you halfway there:
var ErrTooShort = errors.New("too short")
func convertintorec(data []byte) (t *Test, err error) {
t = &Test{}
if len(data) < 9 {
err = ErrTooShort
return
}
uniqnum := string(data[:9])
t.uniqnum, err = strconv.ParseInt(uniqnum, 10, 64)
if err != nil {
err = fmt.Errorf("bad uniqnum %q: %w", uniqnum, err)
return
}
data = data[9:]
if len(data) < 32 {
err = fmt.Errorf("gid %w", ErrTooShort)
return
}
t.guid = string(data[:32])
// t.flag / t.printnum is left to implement...
return
}
https://play.golang.org/p/ZPzWC2IYjj1

Golang JSON Array

I am trying to use WordNik () to get random words for the dictionary for this script: https://github.com/jmagrippis/password
WordNik is outputting:
[{"id":7936915,"word":"Tanganyikan"},{"id":27180,"word":"cartographic"},{"id":48094,"word":"deterministic"},{"id":1485119,"word":"higher-risk"},{"id":120986,"word":"juristic"},{"id":1830806,"word":"magnetorheological"},{"id":320495,"word":"quelled"},{"id":324610,"word":"remoter"},{"id":215158,"word":"telemetric"},{"id":225207,"word":"uninquisitive"}]
Here is my code:
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
"time"
"github.com/jmagrippis/password"
)
type Words struct {
id []int64
word []string
}
type GetWordsResponse struct {
WordList []Words
}
func getWords(speech string) (*GetWordsResponse, error) {
url := fmt.Sprintf("http://api.wordnik.com/v4/words.json/randomWords?hasDictionaryDef=false&includePartOfSpeech=%s&minCorpusCount=0&maxCorpusCount=-1&minDictionaryCount=1&maxDictionaryCount=-1&minLength=5&maxLength=-1&limit=10&api_key=api_key", speech)
res, err := http.Get(url)
if err != nil {
panic(err.Error())
}
body, err := ioutil.ReadAll(res.Body)
if err != nil {
panic(err.Error())
}
var s = new(GetWordsResponse)
var arr []string
_ = json.Unmarshal([]byte(body), &arr)
log.Printf("Unmarshaled: %v", arr)
return s, err
}
func main() {
dictionary := &password.Dictionary{
Adjectives: []string{"beautiful", "homely", "magical", "posh", "excellent"},
Subjects: []string{"mermaids", "unicorns", "lions", "piranhas"},
Verbs: []string{"love", "fancy", "eat", "bring", "fear", "aggravate"},
Adverbs: []string{"cuddling", "slapping", "shouting", "jumping"},
Objects: []string{"teddy-bears", "diamonds", "buckets", "boxes"},
}
generator := password.NewGenerator(dictionary, time.Now().UnixNano())
pass := generator.Generate()
fmt.Printf("%s", pass)
getWords("Verb")
}
As you can see, what I am trying to do is use WordNik API to request Adverbs, nouns, etc and then make a dictionary based off those words to generate a password. I am horrible with arrays and handling data.
As pointed out in the comments you need to export the fields so they can be unmarshaled
The encoding/json package relies on reflection and since it is in another package, it cannot access unexported fields. ( in go, fields, methods or functions that start with a small letter are unexported whereas with capital letters they are exported )
Then your example json does not contain the WordList at all, so what you want is unmarshaling directly into the array of Words. Also a words object only consists of Id and Word and not of arrays themselves.
type Words struct {
Id int64
Word string
}
func main() {
....
var words []Words
// you don't need to read the whole body first, you can decode in the same turn like this
err := json.NewDecoder(req.Body).Decode(&words)
if nil != err {
log.Fatal(err)
}
...
}
Another very important thing is that you should not ignore errors. This would have helped you to debug the issue. ( What I mean is _ = json.Unmarshal)
As for beginning with go, you could implement a simple test to see if your code works as intended.
https://play.golang.org/p/nuz9uXdka5S < check this working example for reference.

Output of GET request different to view source

I'm trying to extract match data from whoscored.com. When I view the source on firefox, I find on line 816 a big json string with the data I want for that matchid. My goal is to eventually get this json.
In doing this, I've tried to download every page of https://www.whoscored.com/Matches/ID/Live where ID is the id of the match. I wrote a little Go program to GET request each ID up to a certain point:
package main
import (
"fmt"
"io/ioutil"
"net/http"
"os"
)
// http://www.whoscored.com/Matches/614052/Live is the match for
// Eveton vs Manchester
const match_address = "http://www.whoscored.com/Matches/"
// the max id we get
const max_id = 300
const num_workers = 10
// function that get the bytes of the match id from the website
func match_fetch(matchid int) {
url := fmt.Sprintf("%s%d/Live", match_address, matchid)
resp, err := http.Get(url)
if err != nil {
fmt.Println(err)
return
}
// if we sucessfully got a response, store the
// body in memory
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println(err)
return
}
// write the body to memory
pwd, _ := os.Getwd()
filepath := fmt.Sprintf("%s/match_data/%d", pwd, matchid)
err = ioutil.WriteFile(filepath, body, 0644)
if err != nil {
fmt.Println(err)
return
}
}
// data type to send to the workers,
// last means this job is the last one
// matchid is the match id to be fetched
// a matchid of -1 means don't fetch a match
type job struct {
last bool
matchid int
}
func create_worker(jobs chan job) {
for {
next_job := <-jobs
if next_job.matchid != -1 {
match_fetch(next_job.matchid)
}
if next_job.last {
return
}
}
}
func main() {
// do the eveton match as a reference
match_fetch(614052)
var joblist [num_workers]chan job
var v int
for i := 0; i < num_workers; i++ {
job_chan := make(chan job)
joblist[i] = job_chan
go create_worker(job_chan)
}
for i := 0; i < max_id; i = i + num_workers {
for index, c := range joblist {
if i+index < max_id {
v = i + index
} else {
v = -1
}
c <- job{false, v}
}
}
for _, c := range joblist {
c <- job{true, -1}
}
}
The code seems to work in that it fills a directory called match_data with html. The problem is that this html is completely different to what I get in the browser! Here is the section which I think does this: (from the body of the GET request of http://www.whoscored.com/Matches/614052/Live.
(function() {
var z="";var b="7472797B766172207868723B76617220743D6E6577204461746528292E67657454696D6528293B766172207374617475733D227374617274223B7661722074696D696E673D6E65772041727261792833293B77696E646F772E6F6E756E6C6F61643D66756E6374696F6E28297B74696D696E675B325D3D22723A222B286E6577204461746528292E67657454696D6528292D74293B646F63756D656E742E637265617465456C656D656E742822696D6722292E7372633D222F5F496E63617073756C615F5265736F757263653F4553324C555243543D363726743D373826643D222B656E636F6465555249436F6D706F6E656E74287374617475732B222028222B74696D696E672E6A6F696E28292B222922297D3B69662877696E646F772E584D4C4874747052657175657374297B7868723D6E657720584D4C48747470526571756573747D656C73657B7868723D6E657720416374697665584F626A65637428224D6963726F736F66742E584D4C4854545022297D7868722E6F6E726561647973746174656368616E67653D66756E6374696F6E28297B737769746368287868722E72656164795374617465297B6361736520303A7374617475733D6E6577204461746528292E67657454696D6528292D742B223A2072657175657374206E6F7420696E697469616C697A656420223B627265616B3B6361736520313A7374617475733D6E6577204461746528292E67657454696D6528292D742B223A2073657276657220636F6E6E656374696F6E2065737461626C6973686564223B627265616B3B6361736520323A7374617475733D6E6577204461746528292E67657454696D6528292D742B223A2072657175657374207265636569766564223B627265616B3B6361736520333A7374617475733D6E6577204461746528292E67657454696D6528292D742B223A2070726F63657373696E672072657175657374223B627265616B3B6361736520343A7374617475733D22636F6D706C657465223B74696D696E675B315D3D22633A222B286E6577204461746528292E67657454696D6528292D74293B6966287868722E7374617475733D3D323030297B706172656E742E6C6F636174696F6E2E72656C6F616428297D627265616B7D7D3B74696D696E675B305D3D22733A222B286E6577204461746528292E67657454696D6528292D74293B7868722E6F70656E2822474554222C222F5F496E63617073756C615F5265736F757263653F535748414E45444C3D313536343032333530343538313538333938362C31373139363833393832313930303534313833392C31333935303737313737393531363432383234342C3132363636222C66616C7365293B7868722E73656E64286E756C6C297D63617463682863297B7374617475732B3D6E6577204461746528292E67657454696D6528292D742B2220696E6361705F6578633A20222B633B646F63756D656E742E637265617465456C656D656E742822696D6722292E7372633D222F5F496E63617073756C615F5265736F757263653F4553324C555243543D363726743D373826643D222B656E636F6465555249436F6D706F6E656E74287374617475732B222028222B74696D696E672E6A6F696E28292B222922297D3B";for (var i=0;i<b.length;i+=2){z=z+parseInt(b.substring(i, i+2), 16)+",";}z = z.substring(0,z.length-1); eval(eval('String.fromCharCode('+z+')'));})();
The reason I think this is the case is that the javascript in the page fetches and edits the DOM to what I see on view source. How can I get golang to run the javascript? Is there are library to do this? Better still, could I directly grab the JSON from the servers?
This can be done with https://godoc.org/github.com/sourcegraph/webloop#View.EvaluateJavaScript
Read their main example https://github.com/sourcegraph/webloop
What you need is a "headless browser" in general.
In general it is better to use an Web API vs. scraping. For example, whoscored themselves use OPTA which you should be able to access directly.
http://www.jokecamp.com/blog/guide-to-football-and-soccer-data-and-apis/#opta

Unable to Unmarshall the data correctly in golang from a map to a structure

I am currently unable to unmarshall the data correctly from a map into a structure. Following is a code snippet (Brief Code at playground):
Request you to kindly provide the reason of getting default values on unmarshlling back the data.
package main
import (
"fmt"
"encoding/json"
"os"
)
func main() {
fmt.Println("Hello, playground")
type PDPOffer struct {
cart_value int `json:"cart_value"`
discount_amount_default int `json:"discount_amount_default"`
max_discount string `json:"max_discount"`
}
a:= map[string]interface{} {
"cart_value" : 1,
"max_discount" : 2,
}
var pdf PDPOffer
b, err := json.Marshal(a)
if err != nil {
fmt.Println("error:", err)
}
os.Stdout.Write(b)//working
err1 := json.Unmarshal(b, &pdf)
if err1 != nil {
fmt.Println("error:", err)
}
fmt.Printf("%+v", pdf)//displaying just the defualt values????????
}
json.Marshal and json.Unmarshal can only work on exported struct fields. Your fields aren't exported and aren't visible to the json code.
The reason for your unmarshal not working is that you need to expose fields of the struct and for doing that you need to start field name with Capital letters. Something has below:
type PDPOffer struct {
Cart_value int `json:"cart_value"`
Discount_amount_default int `json:"discount_amount_default"`
Max_discount string `json:"max_discount"`
}
Also, you were trying to marshal an int value into a string for "max_discount", you need to store it as a string in your map that you're marshaling:
a := map[string]interface{}{
"cart_value": 1,
"max_discount": "2",
}
The error handling had a bug checking for err1 != nil then printing err which was hiding the message error: json: cannot unmarshal number into Go value of type string
Working example with all the fixes: http://play.golang.org/p/L8VC-531nS

how to access deeply nested json keys and values

I'm writing a websocket client in Go. I'm receiving the following JSON from the server:
{"args":[{"time":"2013-05-21 16:57:17"}],"name":"send:time"}
I'm trying to access the time parameter, but just can't grasp how to reach deep into an interface type:
package main;
import "encoding/json"
import "log"
func main() {
msg := `{"args":[{"time":"2013-05-21 16:56:16", "tzs":[{"name":"GMT"}]}],"name":"send:time"}`
u := map[string]interface{}{}
err := json.Unmarshal([]byte(msg), &u)
if err != nil {
panic(err)
}
args := u["args"]
log.Println( args[0]["time"] ) // invalid notation...
}
Which obviously errors, since the notation is not right:
invalid operation: args[0] (index of type interface {})
I just can't find a way to dig into the map to grab deeply nested keys and values.
Once I can get over grabbing dynamic values, I'd like to declare these messages. How would I write a type struct to represent such complex data structs?
You may like to consider the package github.com/bitly/go-simplejson
See the doc: http://godoc.org/github.com/bitly/go-simplejson
Example:
time, err := json.Get("args").GetIndex(0).String("time")
if err != nil {
panic(err)
}
log.Println(time)
The interface{} part of the map[string]interface{} you decode into will match the type of that field. So in this case:
args.([]interface{})[0].(map[string]interface{})["time"].(string)
should return "2013-05-21 16:56:16"
However, if you know the structure of the JSON, you should try defining a struct that matches that structure and unmarshal into that. Ex:
type Time struct {
Time time.Time `json:"time"`
Timezone []TZStruct `json:"tzs"` // obv. you need to define TZStruct as well
Name string `json:"name"`
}
type TimeResponse struct {
Args []Time `json:"args"`
}
var t TimeResponse
json.Unmarshal(msg, &t)
That may not be perfect, but should give you the idea
I'm extremely new to Golang coming from Python, and have always struggled with encode/decoding json. I found gjson at https://github.com/tidwall/gjson, and it helped me immensely:
package main
import "github.com/tidwall/gjson"
func main() {
msg := (`{"args":[{"time":"2013-05-21 16:56:16", "tzs":[{"name":"GMT"}]}],"name":"send:time"}`)
value := gjson.Get(msg, "args.#.time")
println(value.String())
}
-----------------------
["2013-05-21 16:56:16"]
Additionally, I noticed the comment of how to convert into Struct
package main
import (
"encoding/json"
"fmt"
)
type msgFormat struct {
Time string `json:"time"`
Tzs msgFormatTzs `json:"tzs"`
Name string `json:"name"`
}
type msgFormatTzs struct {
TzsName string `json:"name"`
}
func main() {
msg := (`{"args":[{"time":"2013-05-21 16:56:16", "tzs":[{"name":"GMT"}]}],"name":"send:time"}`)
r, err := json.Marshal(msgFormatTzs{msg})
if err != nil {
panic(err)
}
fmt.Printf("%v", r)
}
Try on Go playground

Resources