How to common function of package model in golang? - go

I have 2 package as model:
class:
package class
import (
"encoding/json"
"student_management/model/base"
)
type Classes struct {
Classes []Class
}
type Class struct {
Id int `json:"student_id"`
Name int `json:"name"`
HomeroomTeacherId int `json:"homeroom_teacher_id"`
}
func ReadData() (chan Class, int) {
var classes Classes
byteValue := base.ReadJSON("db/student_class.json")
json.Unmarshal(byteValue, &classes)
classChannel := make(chan Class)
go func () {
for i := 0; i < len(classes.Classes); i++ {
classChannel <- classes.Classes[i]
}
close(classChannel)
}()
return classChannel, len(classes.Classes)
}
teacher:
package teacher
import (
"encoding/json"
base "student_management/model/base"
)
type Teachers struct {
Teachers []Teacher `json:"teachers"`
}
type Teacher struct {
base.Persions
HomeroomTeacher bool `json:"homeroom_teacher"`
}
func ReadData() (chan Teacher, int) {
var teachers Teachers
byteValue := base.ReadJSON("db/teachers.json")
json.Unmarshal(byteValue, &teachers)
teacherChannel := make(chan Teacher)
go func () {
for i := 0; i < len(teachers.Teachers); i++ {
teacherChannel <- teachers.Teachers[i]
}
close(teacherChannel)
}()
return teacherChannel, len(teachers.Teachers)
}
So you can see the ReadData function being repeated. And now I can use class.ReadData() and teacher.ReadData() to call data from channel.
How can I write ReadData() function once for both packages to use?
I tried creating a base package use generics like this:
package base
func ReadData[Models any, Model any](fileName string, m Models) (chan interface{}, int) {
byteValue := ReadJSON(fileName)
json.Unmarshal(byteValue, &m)
channel := make(chan Model)
go func () {
for i := 0; i < len(m.Models); i++ {
channel <- m.Models[i]
}
close(channel)
}()
return channel, len(models.Models)
}
but m.Models not found, i mean teachers.Teachers or classes.Classes can not be used
Please tell me what to do in this case

Use generics (introduced in Go 1.18). Create a single ReadData() function, use a parameter type denoting the values you want to decode from JSON and deliver on the channel.
Note: you should check for errors and report them (including from base.ReadJSON()).
func ReadData[T any](fileName, fieldName string) (chan T, int, error) {
var m map[string][]T
byteValue := base.ReadJSON(fileName)
if err := json.Unmarshal(byteValue, &wrapper); err != nil {
return nil, 0, err
}
values := m[fieldName]
valuesChannel := make(chan T)
go func() {
for _, v := range values {
valuesChannel <- v
}
close(valuesChannel)
}()
return valuesChannel, len(values), nil
}
Example calling it:
ch, n, err := ReadData[class.Class]("db/student_class.json", "classes")
// or
ch, n, err := ReadData[teacher.Teacher]("db/teachers.json", "teachers")
Note that it should be redundant to return the number of read values. Since you properly close the returned channel, the caller can use a for range over the returned channel which will receive all values sent on it properly, and then terminate.
Also note that since all values are ready (decoded) when you return the channel, this concurrency is redundant and just makes things less efficient. You have a slice of the decoded values, just return it and let the caller choose how it wishes to process it (concurrently or not).
So your ReadData() should look like this:
func ReadData[T any](fileName, fieldName string) ([]T, error) {
var m map[string][]T
byteValue := base.ReadJSON(fileName)
if err := json.Unmarshal(byteValue, &wrapper); err != nil {
return nil, err
}
return m[fieldName]
}
Also note that if the input JSON object has a single field, it's not necessary to pass the fieldName, you can get the value from the decoded m map like this:
func ReadData[T any](fileName string) ([]T, error) {
var m map[string][]T
byteValue := base.ReadJSON(fileName)
if err := json.Unmarshal(byteValue, &wrapper); err != nil {
return nil, err
}
for _, v := range m {
return v, nil
}
return nil, errors.New("empty JSON")
}
And then calling it is simply:
classes, err := ReadData[class.Class]("db/student_class.json")
// or
teachers, err := ReadData[teacher.Teacher]("db/teachers.json")

Related

a problem about all goroutines are asleep, a demo of observer pattern

the demo is over here, I was tring to use channel to achieve observer pattern, but a fatal error was encountered.
If I use a simple channel in the test, everyting is going well.
Maybe the observer pattern in go shouldn`t like this.
I would appreicate if anybody can help, thank you so much~~~
package observer
import "reflect"
type ISubject interface {
Register(observer IObserver) error
CancelRegister(observer IObserver) error
Publish() error
}
type Subject struct {
RegisterChannels []chan string `json:"registerChannels"`
}
func (s *Subject) Register(observer Observer) error {
observer.ListenChannel = make(chan string)
s.RegisterChannels = append(s.RegisterChannels, observer.ListenChannel)
return nil
}
func (s *Subject) CancelRegister(observer Observer) error {
for index, val := range s.RegisterChannels {
if reflect.DeepEqual(val, observer.ListenChannel) {
s.RegisterChannels = append(s.RegisterChannels[:index], s.RegisterChannels[index+1:]...)
}
}
return nil
}
func (s *Subject) Publish(msg string) error {
for i := 0; i < len(s.RegisterChannels); i++ {
s.RegisterChannels[i] <- msg
}
return nil
}
package observer
import "fmt"
type IObserver interface {
Subscribe()
}
type Observer struct {
ListenChannel chan string `json:"listenChannel"`
}
func (observer Observer) Subscribe() {
msg := <-observer.ListenChannel
fmt.Println(msg)
}
func Test_Sub(t *testing.T) {
s1 := Subject{}
o1 := Observer{}
err := s1.Register(o1)
if err != nil {
return
}
go func() {
o1.Subscribe()
fmt.Println(1)
}()
fmt.Println(2)
err = s1.Publish("hahaha")
if err != nil {
return
}
}
You must pass observer object by reference (not by value) or you should make the observer ListenChannel in outside of Register (in Test_Sub func in your test).
By Reference like as:
func (s *Subject) Register(observer Observer) error {
s.RegisterChannels = append(s.RegisterChannels, observer.ListenChannel)
return nil
}
func Test_Sub(t *testing.T) {
s1 := Subject{}
o1 := Observer{}
err := s1.Register(&o1)
...
}
or makeing in outside of Register func like as:
func (s *Subject) Register(observer Observer) error {
s.RegisterChannels = append(s.RegisterChannels, observer.ListenChannel)
return nil
}
func Test_Sub(t *testing.T) {
s1 := Subject{}
o1 := Observer{}
o1.ListenChannel = make(chan string)
err := s1.Register(o1)
...
}
Also in your Publish func, I think that better write to the channel with a go routine,
because if a observer couldn't read its channel It will lock all other observer in a subscription
func (s *Subject) Publish(msg string) error {
for i := 0; i < len(s.RegisterChannels); i++ {
go func(i int) { s.RegisterChannels[i] <- msg }(i)
}
return nil
}

Go: Implementing a ManyDecode for a "set" of individual results

I have implemented a very simple Decode method (using gob.Decoder for now) - this works well for single responses - it would even work well for slices, but I need to implement a DecodeMany method where it is able to decode a set of individual responses (not a slice).
Working Decode method:
var v MyType
_ = Decode(&v)
...
func Decode(v interface{}) error {
buf, _ := DoSomething() // func DoSomething() ([]byte, error)
// error handling omitted for brevity
return gob.NewDecoder(bytes.NewReader(buf)).Decode(v)
}
What I'm trying to do for a DecodeMany method is to deal with a response that isn't necessarily a slice:
var vv []MyType
_ = DecodeMany(&vv)
...
func DecodeMany(vv []interface{}) error {
for _, g := range DoSomething() { // func DoSomething() []struct{Buf []bytes}
// Use g.Buf as an individual "interface{}"
// want something like:
var v interface{} /* Somehow create instance of single vv type? */
_ = gob.NewDecoder(bytes.NewReader(g.Buf)).Decode(v)
vv = append(vv, v)
}
return
}
Besides not compiling the above also has the error of:
cannot use &vv (value of type *[]MyType) as type []interface{} in argument to DecodeMany
If you want to modify the passed slice, it must be a pointer, else you must return a new slice. Also if the function is declared to have a param of type []interface{}, you can only pass a value of type []interface{} and no other slice types... Unless you use generics...
This is a perfect example to start using generics introduced in Go 1.18.
Change DecodeMany() to be generic, having a T type parameter being the slice element type:
When taking a pointer
func DecodeMany[T any](vv *[]T) error {
for _, g := range DoSomething() {
var v T
if err := gob.NewDecoder(bytes.NewReader(g.Buf)).Decode(&v); err != nil {
return err
}
*vv = append(*vv, v)
}
return nil
}
Here's a simple app to test it:
type MyType struct {
S int64
}
func main() {
var vv []MyType
if err := DecodeMany(&vv); err != nil {
panic(err)
}
fmt.Println(vv)
}
func DoSomething() (result []struct{ Buf []byte }) {
for i := 3; i < 6; i++ {
buf := &bytes.Buffer{}
v := MyType{S: int64(i)}
if err := gob.NewEncoder(buf).Encode(v); err != nil {
panic(err)
}
result = append(result, struct{ Buf []byte }{buf.Bytes()})
}
return
}
This outputs (try it on the Go Playground):
[{3} {4} {5}]
When returning a slice
If you choose to return the slice, you don't have to pass anything, but you need to assign the result:
func DecodeMany[T any]() ([]T, error) {
var result []T
for _, g := range DoSomething() {
var v T
if err := gob.NewDecoder(bytes.NewReader(g.Buf)).Decode(&v); err != nil {
return result, err
}
result = append(result, v)
}
return result, nil
}
Using it:
vv, err := DecodeMany[MyType]()
if err != nil {
panic(err)
}
fmt.Println(vv)
Try this one on the Go Playground.

How to retype a function in Golang?

In my current Golang project, I use buffering of logs before sending them to Elasticsearch within my log library. I want to introduce something like C atexit() function to flush all pending logs in case of an unexpected exit.
I have found atexit library but it was insufficient in my case because it does not allow passing of arguments to handler functions. I decided to write my version and ended up with roughly similar structure. I have module atexit:
package atexit
import (
"fmt"
"os"
"reflect"
)
var handlers []interface{}
var params []interface{}
func runHandler(handler interface{}, p interface{}) {
defer func() {
if err := recover(); err != nil {
fmt.Fprintln(os.Stderr, "error: atexit handler error:", err)
}
}()
f := reflect.ValueOf(handler)
s := reflectSlice(p)
fmt.Printf("%#v", s)
f.Call(s)
}
func reflectSlice(slice interface{}) []reflect.Value {
s := reflect.ValueOf(slice)
if s.Kind() != reflect.Slice {
panic("InterfaceSlice() given a non-slice type")
}
ret := make([]reflect.Value, s.Len())
for i:=0; i<s.Len(); i++ {
ret[i] = reflect.ValueOf(s.Index(i))
}
return ret
}
func runHandlers() {
for i, handler := range handlers {
runHandler(handler, params[i])
}
}
func Exit(code int) {
runHandlers()
os.Exit(code)
}
func Register(handler interface{}, p interface{}) {
f := reflect.TypeOf(handler)
if f.Kind() != reflect.Func {
panic("Register() given a non-function type")
}
handlers = append(handlers, handler)
params = append(params, p)
}
and I am calling it from the main program:
package main
import (
"fmt"
"./atexit"
"encoding/json"
"reflect"
)
type Batch struct {
Index string `json:"index"`
Type string `json:"_Type"`
Content interface{} `json:"Content"`
}
func flush(b ...interface{}) {
for _, entry := range(b) {
fmt.Printf("%#v\n", entry)
fmt.Println(reflect.TypeOf(reflect.ValueOf(entry)))
fmt.Println(reflect.TypeOf(entry))
a, err := json.Marshal(entry)
if err != nil {
fmt.Println("error!")
}
fmt.Println(string(a))
}
}
func handler(v ...interface{}) {
fmt.Println("Exiting")
for _, batch := range(v) {
fmt.Printf("%#v\n", batch)
}
}
func main() {
type ColorGroup struct {
ID int
Name string
Colors []string
}
group := ColorGroup{
ID: 1,
Name: "Reds",
Colors: []string{"Crimson", "Red", "Ruby", "Maroon"},
}
fmt.Printf("%#v\n", group)
r, err := json.Marshal(group)
if err != nil {
fmt.Println("error:", err)
}
fmt.Println(string(r))
b := []Batch{Batch{"index", "type", "content1"},Batch{"index", "type", "content2"}}
atexit.Register(handler, b)
atexit.Register(flush, []ColorGroup{group})
atexit.Exit(0)
}
As you can see, by calling reflect.ValueOf() I get structure reflect.Value which is then passed to callback function. The problem seems to be that this structure does not contain metadata about json export or is not handled correctly with json.Marshal(), which then outputs empty json. Is there any way I can pass correct []interface{} structure to the callback function or some similar mechanism which would do roughly what I'm trying to accomplish? Please note that I want a general callback mechanism which should be independent of the type passed to it. So far it seems to be impossible or at least very limited by f.Call(s), which is called in runHandler() and takes reflect.Value as its parameter.
You should not be using reflection for this. You should make your handlers be of type func(), then just pass in an anonymous function (possibly a closure) that handles any necessary arguments directly. Example:
atexit.Register(func() {
handler(b)
})

Is it a better way to do parallel programming that this?

I made this script for getting the follower count of "influencers" from instagram
the "runtime" number I am getting from it is between 550-750ms.
It is not that bad, but I am wondering whether it could be better or not (as I am a golang noob - learning it 3 weeks only)
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
"sync"
"time"
)
type user struct {
User userData `json:"user"`
}
type userData struct {
Followers count `json:"followed_by"`
}
type count struct {
Count int `json:"count"`
}
func getFollowerCount(in <-chan string) <-chan int {
out := make(chan int)
go func() {
for un := range in {
URL := "https://www.instagram.com/" + un + "/?__a=1"
resp, err := http.Get(URL)
if err != nil {
// handle error
fmt.Println(err)
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
var u user
err = json.Unmarshal(body, &u)
if err != nil {
fmt.Println(err)
}
// return u.User.Followers.Count
out <- u.User.Followers.Count
}
close(out)
}()
return out
}
func merge(cs ...<-chan int) <-chan int {
var wg sync.WaitGroup
out := make(chan int)
output := func(c <-chan int) {
for n := range c {
out <- n
}
wg.Done()
}
wg.Add(len(cs))
for _, c := range cs {
go output(c)
}
go func() {
wg.Wait()
close(out)
}()
return out
}
func gen(users ...string) <-chan string {
out := make(chan string)
go func() {
for _, u := range users {
out <- u
}
close(out)
}()
return out
}
func main() {
start := time.Now()
fmt.Println("STARTING UP")
usrs := []string{"kanywest", "kimkardashian", "groovyq", "kendricklamar", "barackobama", "asaprocky", "champagnepapi", "eminem", "drdre", "g_eazy", "skrillex"}
in := gen(usrs...)
d1 := getFollowerCount(in)
d2 := getFollowerCount(in)
d3 := getFollowerCount(in)
d4 := getFollowerCount(in)
d5 := getFollowerCount(in)
d6 := getFollowerCount(in)
d7 := getFollowerCount(in)
d8 := getFollowerCount(in)
d9 := getFollowerCount(in)
d10 := getFollowerCount(in)
for d := range merge(d1, d2, d3, d4, d5, d6, d7, d8, d9, d10) {
fmt.Println(d)
}
elapsed := time.Since(start)
log.Println("runtime", elapsed)
}
I agree with jeevatkm, there are numerous way to implement your task and improve it. Some notes:
Separate the function that actually do the job (i.e. fetch result from remote service) and the function which is responsible for coordinating all the jobs.
It is a good practice to propagate an errorto the caller instead of consumes (handles) it in a function to be called.
Since the jobs are done in parallel, the result could be returned in undetermined order. Thus, besides follower count, result should contains other related information(s).
The following implementation may be one alternative:
package main
import (
"encoding/json"
"errors"
"fmt"
"net/http"
"sync"
"time"
)
type user struct {
User userData `json:"user"`
}
type userData struct {
Followers count `json:"followed_by"`
}
type count struct {
Count int `json:"count"`
}
//Wrap username, count, and error. See (3) above.
type follower struct {
Username string
Count int
Error error
}
//GetFollowerCountFunc is a function for
//fetching follower count of a specific user.
type GetFollowerCountFunc func(string) (int, error)
//Mockup function for test
func mockGetFollowerCountFor(userName string) (int, error) {
if len(userName) < 9 {
return -1, errors.New("mocking error in get follower count")
}
return 10, nil
}
//Fetch result from remote service. See (1) above.
func getFollowerCountFor(userName string) (int, error) {
URL := "https://www.instagram.com/" + userName + "/?__a=1"
resp, err := http.Get(URL)
if err != nil {
return -1, err
}
defer resp.Body.Close()
var u user
if err := json.NewDecoder(resp.Body).Decode(&u); err != nil {
return -1, err
}
return u.User.Followers.Count, nil
}
//Function that coordinates/distributes the jobs. See (1), (2) above.
func getFollowersAsync(users []string, fn GetFollowerCountFunc) <-chan follower {
//allocate channels for storing result
//number of allocated channels define the maximum *parallel* worker
followers := make(chan follower, len(users))
//The following is also valid
//followers := make(chan follower, 5)
//Do the job distribution in goroutine (Asynchronously)
go func() {
var wg sync.WaitGroup
wg.Add(len(users))
for _, u := range users {
//Run a *parallel* worker
go func(uid string) {
cnt, err := fn(uid)
if err != nil {
followers <- follower{uid, -1, err}
} else {
followers <- follower{uid, cnt, nil}
}
wg.Done()
}(u)
}
//wait all workers finish
wg.Wait()
//close the channels so the `for ... range` will exit gracefully
close(followers)
}()
//This function will returns immediately
return followers
}
func main() {
start := time.Now()
fmt.Println("STARTING UP")
usrs := []string{"kanywest", "kimkardashian", "groovyq", "kendricklamar", "barackobama", "asaprocky", "champagnepapi", "eminem", "drdre", "g_eazy", "skrillex"}
results := getFollowersAsync(usrs, getFollowerCountFor)
//For TESTING:
//results := getFollowersAsync(usrs, mockGetFollowerCountFor)
for r := range results {
if r.Error != nil {
fmt.Printf("Error for user '%s' => %v", r.Username, r.Error)
} else {
fmt.Printf("%s: %d\n", r.Username, r.Count)
}
}
elapsed := time.Since(start)
fmt.Println("runtime", elapsed)
}
Welcome to Go, happy learning.
You're doing good, you can improve your program many ways (such as json decoder, less no of chan, etc). Following is one of the approach. Execution time is between 352-446ms (take it with grain of salt, since network call is involved in your code. Might vary based on server response time).
Your updated code:
package main
import (
"encoding/json"
"fmt"
"log"
"net/http"
"sync"
"time"
)
type user struct {
User userData `json:"user"`
}
type userData struct {
Followers count `json:"followed_by"`
}
type count struct {
Count int `json:"count"`
}
func getFollowerCount(username string, result chan<- int, wg *sync.WaitGroup) {
defer wg.Done()
reqURL := "https://www.instagram.com/" + username + "/?__a=1"
resp, err := http.Get(reqURL)
if err != nil {
log.Println(err)
return
}
defer resp.Body.Close()
var u user
if err := json.NewDecoder(resp.Body).Decode(&u); err != nil {
log.Println(err)
return
}
result <- u.User.Followers.Count
}
func execute(users []string, result chan<- int) {
wg := &sync.WaitGroup{}
for _, username := range users {
wg.Add(1)
go getFollowerCount(username, result, wg)
}
wg.Wait()
result <- -1
}
func main() {
start := time.Now()
fmt.Println("STARTING UP")
usrs := []string{"kanywest", "kimkardashian", "groovyq", "kendricklamar", "barackobama", "asaprocky", "champagnepapi", "eminem", "drdre", "g_eazy", "skrillex"}
result := make(chan int)
go execute(usrs, result)
for v := range result {
if v == -1 {
break
}
fmt.Println(v)
}
elapsed := time.Since(start)
fmt.Println("runtime:", elapsed)
}

How to return a error from a goroutine (through channels)

When i write a function in Go, it should return a value and an error like
func createHashedPassword(password string) (string, error) {
//code
}
I want to execute this createHashedPassword in a goroutine and I think to pass data via channel.
But my question is, how can I handle error here or in the goroutine?
It's common to bundle multiple outputs into a struct, and return them together over a single channel.
type Result struct {
Message string
Error error
}
ch := make(chan Result)
You can pass in an error channel as well as a result channel.
errors := make(chan error, 0)
results := make(chan string, 0)
password := "test"
go func() {
result, err := createHashedPassword(string password)
if err != nil {
errors <- err
return
}
results <- result
}()
// do something else
// When you are ready to read from goroutine do this:
select {
case err := <- errors:
println(err)
case res := <- results:
println(res)
}
Here are my two preferred ways:
Two channels, wrapped
This is the "two channels" way, but wrapped into a function to make it look similar to the common pattern:
func createHashedPasswordAsynchronously(password string) (chan string, chan error) {
resultCh := make(chan string)
errorCh := make(chan error)
go func(password string) {
//code
if err != nil {
errorCh <- errors.New("Does not compute")
} else {
resultCh <- "8badf00d"
}
}(password)
return resultCh, errorCh
}
And called like this:
resultCh, errorCh := createHashedPasswordAsynchronously("mysecret")
select {
case result := <-resultCh:
storeHashedPassword(result)
case err := <-errorCh:
log.Println(err.Error())
}
Anonymous struct
This is the "anonymous struct" way, similar to #saward's answer, but without naming the struct members explicitly:
go func(password string, ch chan struct {
string
error
}) {
//code
if err != nil {
ch <- struct {
string
error
}{"", errors.New("Does not compute")}
} else {
ch <- struct {
string
error
}{"8badf00d", nil}
}
}("mysecret", ch)
r := <-ch
if r.error != nil {
log.Println(r.error.Error())
} else {
storeHashedPassword(r.string)
}
(since I cannot comment yet...)
I echo what JimB said with:
type Result struct {
Message string
Error error
}
ch := make(chan Result)
The trouble with two separate channels, one for the result, and another for the error, is that (as I understand) it won't support concurrent threads out of the box.
You could, for example, have two threads sending data at the same time, where the replies get out of order. That is, you receive the result from thread 1 first, but the error from thread 2 first.
It's easy to create new types like JimB suggested, and should work well with goroutines.
One of the common patterns is to pass a error channel and a result channel to the function:
package main
import (
"fmt"
)
func main() {
password := "blah blah"
resultChan := make(chan string)
errChan := make(chan error)
go createHashedPassword(password, resultChan, errChan)
// Do some other stuff
select {
case err := <-errChan:
fmt.Println(err)
case res := <-resultChan:
fmt.Println(res)
}
}
func createHashedPassword(password string, resultChan chan string, errChan chan error) {
// your code for hashing and stuff
if err != nil {
errChan <- err
return
}
resultChan <- result
}
You can also make channels inside the createHashedPassword as AndreKR said,
or call the createHashedPassword inside an anonymous goroutine as Ian Davis said.
There is also some other possible patterns, and you can combine any of them with combined struct:
type Result struct {
err error
msg string
}
or
type Result struct {
err error
msg *string // in cases that empty string can be a true result, you can evaluate msg with nil
}
I should also mention that in some special cases, we don't return the error and just log it there.
package main
import (
"fmt"
"log"
)
func main() {
password := "blah blah"
resultChan := make(chan string)
go createHashedPassword(password, resultChan)
// Do some other stuff
fmt.Println(<-resultChan) // Be careful that it's a blocking code.
}
func createHashedPassword(password string, resultChan chan string) {
// your code for hashing and stuff
if err != nil {
log.Println(err) // *** Here we just log the error, or handle it.
}
resultChan <- result
}

Resources