Pattern for fetching multiple fields in parallel - go

I need to fetch multiple fields in parallel for my system from external services (in this example, simulated by Name(), Age() and CanDrive() methods).
The fetchUser() method does what I want, but it seems too verbose specially if you consider I could have 10+ fields. Are there better ways I can implement this?
playground: https://play.golang.org/p/90sNq1GmrD8
Code (same as in playground):
package main
import (
"fmt"
"sync"
)
type User struct {
Name string
Age int
CanDrive *bool
}
func Name() (string, error) {
return "foobar", nil
}
func Age() (int, error) {
return 25, nil
}
func CanDrive() (bool, error) {
return true, nil
}
func fetchUser() (*User, error) {
var wg sync.WaitGroup
errs := make(chan error)
user := &User{}
wg.Add(1)
go func() {
var err error
defer wg.Done()
user.Name, err = Name()
errs <- err
}()
wg.Add(1)
go func() {
var err error
defer wg.Done()
user.Age, err = Age()
errs <- err
}()
wg.Add(1)
go func() {
defer wg.Done()
canDrive, err := CanDrive()
if err == nil {
user.CanDrive = &canDrive
}
errs <- err
}()
// wait until all go-routines are completed successfully
// if that's the case, close the errs channel
go func() {
wg.Wait()
close(errs)
}()
// keep waiting for errors (or for the error channel to be closed
// if all calls succeed)
for err := range errs {
if err != nil {
return nil, err
}
}
return user, nil
}
func main() {
user, _ := fetchUser()
fmt.Println(user)
}

Without knowing more of the specifics of your scenario, my only suggestion would be to separate out the Go routine error handling into another package.
Fortunately, a package already exists that does the same thing, named errgroup. Below is an implementation of your original code using the errgroup package:
package main
import (
"context"
"fmt"
"golang.org/x/sync/errgroup"
)
type User struct {
Name string
Age int
CanDrive *bool
}
func Name() (string, error) {
return "foobar", nil
}
func Age() (int, error) {
return 25, nil
}
func CanDrive() (bool, error) {
return true, nil
}
func fetchUser(ctx context.Context) (*User, error) {
group, ctx := errgroup.WithContext(ctx)
user := &User{}
group.Go(func() (err error) {
user.Name, err = Name()
return
})
group.Go(func() (err error) {
user.Age, err = Age()
return
})
group.Go(func() error {
canDrive, err := CanDrive()
if err == nil {
user.CanDrive = &canDrive
}
return err
})
if err := group.Wait(); err != nil {
return nil, err
}
return user, nil
}
func main() {
user, err := fetchUser(context.Background())
fmt.Println(user, err)
}

Related

Could not `chan` pass through RPC in Go?

For the following code, get Message via RPC, why its Done field is nil?
Could not chan pass through RPC?
type Message struct {
Text string
Done chan bool
}
type (
GetArgs struct {
Key string
}
GetReply struct {
Message *Message
}
)
// Client
func connect() *rpc.Client {
client, err := rpc.Dial("tcp", ":1234")
if err != nil {
log.Fatal("dialing:", err)
}
return client
}
func get(key string) *Message {
client := connect()
args := GetArgs{key}
reply := GetReply{}
err := client.Call("KV.Get", &args, &reply)
if err != nil {
log.Fatal("error:", err)
}
client.Close()
return reply.Message
}
// Server
type KV struct {
mu sync.Mutex
messages map[string]*Message
}
func server() {
kv := new(KV)
kv.messages = map[string]*Message{}
done := make(chan bool)
kv.messages["Jan"] = &Message{Text: "hello", Done: done}
rpcs := rpc.NewServer()
rpcs.Register(kv)
l, err := net.Listen("tcp", ":1234")
if err != nil {
log.Fatal("error:", err)
}
go func() {
for {
conn, err := l.Accept()
if err == nil {
go rpcs.ServeConn(conn)
} else {
break
}
}
l.Close()
}()
}
func (kv *KV) Get(args *GetArgs, reply *GetReply) error {
kv.mu.Lock()
defer kv.mu.Unlock()
val, ok := kv.messages[args.Key]
if ok {
reply.Message = val
}
return nil
}
func main() {
server()
fmt.Printf("get() -> %v\n", get("Jan"))
// Output:
// get() -> &{hello <nil>}
}
No. RPC arguments can only be those that are serializable by encoding/gob, and chans are not one of them
Functions and channels will not be sent in a gob. Attempting to encode such a value at the top level will fail. A struct field of chan or func type is treated exactly like an unexported field and is ignored.

Go: negative WaitGroup counter

I'm somewhat new to go and am reworking code that I found somewhere else to fit my needs. Because of that, I don't totally understand what is happening here, although I get the general idea.
I'm running a few websocket clients using go routines, but I'm getting an unexpected error that causes the program to crash. My program seems to close one too many threads (excuse me if this is the wrong terminology) when there is an error reading a message from the websocket (check the conn.ReadMessage() func in the readHandler func). Any ideas on how would I work around this issue? I would really appreciate anyone taking the time to look through it. Thanks in advance!
package main
import (
"context"
"fmt"
"os"
"time"
"os/signal"
"syscall"
"sync"
"net/url"
"github.com/gorilla/websocket"
"strconv"
"encoding/json"
"log"
"bytes"
"compress/gzip"
"io/ioutil"
)
// Structs
type Ping struct {
Ping int64 `json:"ping"`
}
type Pong struct {
Pong int64 `json:"pong"`
}
type SubParams struct {
Sub string `json:"sub"`
ID string `json:"id"`
}
func InitSub(subType string, pair string, i int) []byte {
var idInt string = "id" + strconv.Itoa(i)
subStr := "market." + pair + "." + subType
sub := &SubParams{
Sub: subStr,
ID: idInt,
}
out, err := json.MarshalIndent(sub, "", " ")
if err != nil {
log.Println(err);
}
//log.Println(string(out))
return out
}
// main func
func main() {
var server string = "api.huobi.pro"
pairs := []string{"btcusdt", "ethusdt", "ltcusdt"}
comms := make(chan os.Signal, 1)
signal.Notify(comms, os.Interrupt, syscall.SIGTERM)
ctx := context.Background()
ctx, cancel := context.WithCancel(ctx)
var wg sync.WaitGroup
for x, pair := range pairs {
wg.Add(1)
go control(server, "ws", pair, ctx, &wg, x+1)
}
<-comms
cancel()
wg.Wait()
}
func control(server string, path string, pair string, ctx context.Context, wg *sync.WaitGroup, i int) {
fmt.Printf("Started control for %s\n", server)
url := url.URL {
Scheme: "wss",
Host: server,
Path: path,
}
fmt.Println(url.String())
conn, _, err := websocket.DefaultDialer.Dial(url.String(), nil)
if err != nil {
panic(err)
}
subscribe(conn, pair, i)
defer conn.Close()
var localwg sync.WaitGroup
localwg.Add(1)
go readHandler(ctx, conn, &localwg, server)
<- ctx.Done()
localwg.Wait()
wg.Done()
return
}
func readHandler(ctx context.Context, conn *websocket.Conn, wg *sync.WaitGroup, server string) {
for {
select {
case <- ctx.Done():
wg.Done()
return
default:
_, p, err := conn.ReadMessage()
if err != nil {
wg.Done()
fmt.Println(err)
}
r, err := gzip.NewReader(bytes.NewReader(p))
if(err == nil) {
result, err := ioutil.ReadAll(r)
if(err != nil) {
fmt.Println(err)
}
d := string(result)
fmt.Println(d)
var ping Ping
json.Unmarshal([]byte(d), &ping)
if (ping.Ping > 0) {
str := Pong{Pong: ping.Ping}
msg, err := json.Marshal(str)
if (err == nil) {
fmt.Println(string(msg))
conn.WriteMessage(websocket.TextMessage, []byte(msg))
}
}
}
}
}
}
func subscribe(conn *websocket.Conn, pair string, id int) {
sub := string(InitSub("trade.detail", pair, id))
err := conn.WriteMessage(websocket.TextMessage, []byte(sub))
if err != nil {
panic(err)
}
}
Break out of the readHandler loop when the connection fails:
_, p, err := conn.ReadMessage()
if err != nil {
wg.Done()
fmt.Println(err)
return // <--- add this line
}
Without the return, the function spins in a tight loop reading errors until the panic.
Use defer wg.Done() at the beginning of the goroutine to ensure that Done is called exactly once.
func readHandler(ctx context.Context, conn *websocket.Conn, wg *sync.WaitGroup, server string) {
defer wg.Done()
for {
select {
case <-ctx.Done():
return
default:
_, p, err := conn.ReadMessage()
if err != nil {
fmt.Println(err)
return
}
...
Update the control function also.
Because the caller does not execute any code concurrently with readHander, there's no value in running readHandler is a goroutine. Remove all references to wait groups from readHandler and call the function directly: change go readHandler(ctx, conn, &localwg, server) to readHandler(ctx, conn, server).
There are more issues, but this should move you further along.

How to unit test code consuming gcloud storage?

I want to write unit test for the below code
package main
import (
"context"
"google.golang.org/api/option"
"cloud.google.com/go/storage"
)
var (
NewClient = storage.NewClient
)
func InitializeClient(ctx context.Context) (*storage.Client, error) {
credFilePath := "Storage credentials path."
// Creates a client.
client, err := NewClient(ctx, option.WithCredentialsFile(credFilePath))
if err != nil {
return nil, err
}
return client, nil
}
func createStorageBucket(ctx context.Context, client *storage.Client, bucketName string) (*storage.BucketHandle, error) {
// Sets your Google Cloud Platform project ID.
projectID := "Some project id"
// Creates a Bucket instance.
bucket := client.Bucket(bucketName)
// Creates the new bucket.
ctx, cancel := context.WithTimeout(ctx, time.Second*10)
defer cancel()
if err := bucket.Create(ctx, projectID, nil); err != nil {
return nil, err
}
return bucket, nil
}
func bucketExists(ctx context.Context, client *storage.Client, bucketName string) error {
bucket := client.Bucket(bucketName)
if _, err := bucket.Attrs(ctx); err != nil {
//try creating the bucket
if _, err := createStorageBucket(ctx, client, bucketName); err != nil {
return err
}
}
return nil
}
func main() {
ctx = context.Background()
client, err := InitializeClient(ctx)
bucketName := "Some bucket name"
err = bucketExists(ctx, client, bucketName)
}
bucket.Create() and bucket.Attrs() are http calls, also Bucket(), Object() and NewReader() returning structs(So in my sense there is no meaning of implement interface for this use case)
Note: storage.NewClient() is also http call but i am avoiding external call using monkey pathching approch in my test by providing custom implementaion.
var (
NewClient = storage.NewClient
)
The code is so thin that is is hard to figure how to test that.
I guess Flimzy grasped that reading at the title.
There is a fundamental misunderstanding because of/in the title How to write unit test for gcloud storage?.
Well we should not. They did it. A better title would be How to unit test code consuming gcloud storage? I am not trying to be picky here, but explains what i understood trying to solve that question.
So anyways, this whole thing lead me to write less thin code, so that i can test that the code i write, the lines driving the storage, did what we expect it does.
This whole thing is so convoluted and out of tin air that I dont think it will answer your question.
but anyways, if that helps thinking about this difficulty that is already a win.
package main
import (
"context"
"flag"
"fmt"
"testing"
"time"
"cloud.google.com/go/storage"
"google.golang.org/api/option"
)
type client struct {
c *storage.Client
projectID string
}
func New() client {
return client{}
}
func (c *client) Initialize(ctx context.Context, projectID string) error {
credFilePath := "Storage credentials path."
x, err := NewClient(ctx, option.WithCredentialsFile(credFilePath))
if err == nil {
c.c = x
c.projectID = projectID
}
return err
}
func (c client) BucketExists(ctx context.Context, bucketName string) bool {
if c.c == nil {
return nil, fmt.Errorf("not initialized")
}
bucket := c.c.Bucket(bucketName)
err := bucket.Attrs(ctx)
return err == nil
}
func (c client) CreateBucket(ctx context.Context, bucketName string) (*storage.BucketHandle, error) {
if c.c == nil {
return nil, fmt.Errorf("not initialized")
}
bucket := c.c.Bucket(bucketName)
err := bucket.Create(ctx, c.projectID, nil)
if err != nil {
return nil, err
}
return bucket, err
}
func (c client) CreateBucketIfNone(ctx context.Context, bucketName string) (*storage.BucketHandle, error) {
if !c.BucketExists(bucketName) {
return c.CreateBucket(ctx, c.projectID, bucketName)
}
return c.c.Bucket(bucketName), nil
}
type clientStorageProvider interface { // give it a better name..
Initialize(ctx context.Context, projectID string) (err error)
CreateBucketIfNone(ctx context.Context, bucketName string) (*storage.BucketHandle, error)
CreateBucket(ctx context.Context, bucketName string) (*storage.BucketHandle, error)
BucketExists(ctx context.Context, bucketName string) bool
}
func main() {
flag.Parse()
cmd := flag.Arg(0)
projectID := flag.Arg(1)
bucketName := flag.Arg(2)
ctx, cancel := context.WithTimeout(context.Background(), time.Second*10)
defer cancel()
client := New()
if cmd == "create" {
createBucket(ctx, client, projectID, bucketName)
} else {
// ... more
}
}
// this is the part we are going to test.
func createBucket(ctx context.Context, client clientStorageProvider, projectID, bucketName string) error {
err := client.Initialize(ctx, projectID)
if err != nil {
return err
}
return client.CreateBucketIfNone(bucketName)
// maybe we want to apply retry strategy here,
// and test that the retry was done;
}
type clientFaker struct {
initErr error
createErr error
createIfNoneErr error
bucketExistsErr error
}
func (c clientFaker) Initialize(ctx context.Context, projectID string) (err error) {
return c.initErr
}
func (c clientFaker) CreateBucketIfNone(ctx context.Context, bucketName string) (*storage.BucketHandle, error) {
return nil, c.createIfNoneErr
}
func (c clientFaker) CreateBucket(ctx context.Context, bucketName string) (*storage.BucketHandle, error) {
return nil, c.createErr
}
func (c clientFaker) BucketExists(ctx context.Context, bucketName string) bool {
return nil, c.bucketExistsErr
}
func TestCreateBucketWithFailedInit(t *testing.T) {
c := clientFaker{
initErr: fmt.Errorf("failed init"),
}
ctx := context.Background()
err := createBucket(ctx, c, "", "")
if err == nil {
t.Fatalf("should have failed to initialize the bucket")
}
}
// etc...
note that i am not happy having *storage.BucketHandle as a return parameter, too specific, but i had no use of it (i put it here because it was there, otherwise hanging), so it was hard to design something around that.
noteĀ², it might happen my code is not fully compilable. I am having a dependency problem that i don t want to fix now and it prevent me from seeing all errors (it stops too early in the process)

How to explain the go routine as callback function in this example code

I am confusing the following code, I didn't see the 1st go routine has any returns in TimelineItemStream function, and 2nd go routine call the unsub. In Subscriber interface define the unsub callbackup function.
the code is come from git clone https://github.com/nicolasparada/nakama.git
// TimelineItem model
type TimelineItem struct {
ID string `json:"id"`
UserID string `json:"-"`
PostID string `json:"-"`
Post *Post `json:"post,omitempty"`
}
// Subscriber interface.
type Subscriber interface {
Sub(topic string, cb func(data []byte)) (unsub func() error, err error)
}
func (s *Service) TimelineItemStream(ctx context.Context) (<-chan TimelineItem, error) {
uid, ok := ctx.Value(KeyAuthUserID).(string)
if !ok {
return nil, ErrUnauthenticated
}
tt := make(chan TimelineItem)
unsub, err := s.pubsub.Sub(timelineTopic(uid), func(data []byte) {
go func(r io.Reader) {
var ti TimelineItem
err := gob.NewDecoder(r).Decode(&ti)
if err != nil {
log.Printf("could not gob decode timeline item: %v\n", err)
return
}
tt <- ti
}(bytes.NewReader(data))
})
if err != nil {
return nil, fmt.Errorf("could not subscribe to timeline: %w", err)
}
go func() {
<-ctx.Done()
if err := unsub(); err != nil {
log.Printf("could not unsubcribe from timeline: %v\n", err)
// don't return
}
close(tt)
}()
return tt, nil
}
In the block of call
unsub, err := s.pubsub.Sub(timelineTopic(uid), func(data []byte) {
go func(r io.Reader) {
var ti TimelineItem
err := gob.NewDecoder(r).Decode(&ti)
if err != nil {
log.Printf("could not gob decode timeline item: %v\n", err)
return
}
tt <- ti
}(bytes.NewReader(data))
})
s.pubsub.Sub returns a callback function, but in this block go routine has not any return , how to explain the block of code

golang sync.WaitGroup never completes

I have the below code that fetches a list of URL's and then conditionally downloads a file and saves it to the filesystem. The files are fetched concurrently and the main goroutine waits for all the files to be fetched. But, the program never exits (and there are no errors) after completing all the requests.
What I think is happening is that somehow the amount of go routines in the WaitGroup is either incremented by too many to begin with (via Add) or not decremented by enough (a Done call is not happening).
Is there something I am obviously doing wrong? How would I inspect how many go routines are presently in the WaitGroup so I can better debug what's happening?
package main
import (
"fmt"
"io"
"io/ioutil"
"net/http"
"os"
"strings"
"sync"
)
func main() {
links := parseLinks()
var wg sync.WaitGroup
for _, url := range links {
if isExcelDocument(url) {
wg.Add(1)
go downloadFromURL(url, wg)
} else {
fmt.Printf("Skipping: %v \n", url)
}
}
wg.Wait()
}
func downloadFromURL(url string, wg sync.WaitGroup) error {
tokens := strings.Split(url, "/")
fileName := tokens[len(tokens)-1]
fmt.Printf("Downloading %v to %v \n", url, fileName)
content, err := os.Create("temp_docs/" + fileName)
if err != nil {
fmt.Printf("Error while creating %v because of %v", fileName, err)
return err
}
resp, err := http.Get(url)
if err != nil {
fmt.Printf("Could not fetch %v because %v", url, err)
return err
}
defer resp.Body.Close()
_, err = io.Copy(content, resp.Body)
if err != nil {
fmt.Printf("Error while saving %v from %v", fileName, url)
return err
}
fmt.Printf("Download complete for %v \n", fileName)
defer wg.Done()
return nil
}
func isExcelDocument(url string) bool {
return strings.HasSuffix(url, ".xlsx") || strings.HasSuffix(url, ".xls")
}
func parseLinks() []string {
linksData, err := ioutil.ReadFile("links.txt")
if err != nil {
fmt.Printf("Trouble reading file: %v", err)
}
links := strings.Split(string(linksData), ", ")
return links
}
There are two problems with this code. First, you have to pass a pointer to the WaitGroup to downloadFromURL(), otherwise the object will be copied and Done() will not be visible in main().
See:
func main() {
...
go downloadFromURL(url, &wg)
...
}
Second, defer wg.Done() should be one of the first statements in downloadFromURL(), otherwise if you return from the function before that statement, it won't get "registered" and won't get called.
func downloadFromURL(url string, wg *sync.WaitGroup) error {
defer wg.Done()
...
}
Arguments in Go are always passed by value. Use a pointer when an argument may be modified. Also, make sure that you always execute wg.Done().For example,
package main
import (
"fmt"
"io"
"io/ioutil"
"net/http"
"os"
"strings"
"sync"
)
func main() {
links := parseLinks()
wg := new(sync.WaitGroup)
for _, url := range links {
if isExcelDocument(url) {
wg.Add(1)
go downloadFromURL(url, wg)
} else {
fmt.Printf("Skipping: %v \n", url)
}
}
wg.Wait()
}
func downloadFromURL(url string, wg *sync.WaitGroup) error {
defer wg.Done()
tokens := strings.Split(url, "/")
fileName := tokens[len(tokens)-1]
fmt.Printf("Downloading %v to %v \n", url, fileName)
content, err := os.Create("temp_docs/" + fileName)
if err != nil {
fmt.Printf("Error while creating %v because of %v", fileName, err)
return err
}
resp, err := http.Get(url)
if err != nil {
fmt.Printf("Could not fetch %v because %v", url, err)
return err
}
defer resp.Body.Close()
_, err = io.Copy(content, resp.Body)
if err != nil {
fmt.Printf("Error while saving %v from %v", fileName, url)
return err
}
fmt.Printf("Download complete for %v \n", fileName)
return nil
}
func isExcelDocument(url string) bool {
return strings.HasSuffix(url, ".xlsx") || strings.HasSuffix(url, ".xls")
}
func parseLinks() []string {
linksData, err := ioutil.ReadFile("links.txt")
if err != nil {
fmt.Printf("Trouble reading file: %v", err)
}
links := strings.Split(string(linksData), ", ")
return links
}
As #Bartosz mentioned, you will need to pass a reference to your WaitGroup object. He did a great job discussing the importance of defer ws.Done()
I like WaitGroup's simplicity. However, I do not like that we need to pass the reference to the goroutine because that would mean that the concurrency logic would be mixed with your business logic.
So I came up with this generic function to solve this problem for me:
// Parallelize parallelizes the function calls
func Parallelize(functions ...func()) {
var waitGroup sync.WaitGroup
waitGroup.Add(len(functions))
defer waitGroup.Wait()
for _, function := range functions {
go func(copy func()) {
defer waitGroup.Done()
copy()
}(function)
}
}
So your example could be solved this way:
func main() {
links := parseLinks()
functions := []func(){}
for _, url := range links {
if isExcelDocument(url) {
function := func(url string){
return func() { downloadFromURL(url) }
}(url)
functions = append(functions, function)
} else {
fmt.Printf("Skipping: %v \n", url)
}
}
Parallelize(functions...)
}
func downloadFromURL(url string) {
...
}
If you would like to use it, you can find it here https://github.com/shomali11/util

Resources