How to assert that the request happened when mocking an API concurrently? - go

Followup question to: How do I guarantee that the request happened correctly when mocking an API?
main.go
package main
import (
"net/http"
)
func SomeFeature(host, a string) {
if a == "foo" {
resp, err := http.Get(host + "/foo")
}
if a == "bar" {
resp, err := http.Get(host + "/baz"))
}
// baz is missing, the test should error!
}
main_test.go
package main
import (
"net/http"
"net/http/httptest"
"testing"
)
func TestSomeFeature(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(200)
}))
testCases := []struct {
name string
variable string
}{
{
name: "test 1",
variable: "foo",
},
{
name: "test 2",
variable: "bar",
},
{
name: "test 3",
variable: "baz",
},
}
for _, tc := range testCases {
tc := tc
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
SomeFeature(server.URL, tc.variable)
// assert that the http call happened somehow?
})
}
}
GO Playground: https://go.dev/play/p/EFanSSzgnbk
How to do I assert that each test case send a request to the mocked server?
How can I assert that a request wasn't sent?
All while keeping the tests parallel/concurrent?

You could create a new server for each test case.
Or you can use channels, specifically a map of channels where the key is the test case's identifier, e.g.
getChans := map[string]chan struct{}{}
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
key := strings.Split(r.URL.Path, "/")[1] // extract channel key from path
go func() { getChans[key] <- struct{}{} }()
w.WriteHeader(200)
}))
Add a channel key field to the test case. This will be added to the host's URL and the handler will then extract the key, as demonstrated above, to get the correct channel. Also add a field to indicate whether http.Get should be called or not:
testCases := []struct {
name string
chkey string
variable string
shouldGet bool
}{
{
name: "test 1",
chkey: "key1"
variable: "foo",
shouldGet: true,
},
// ...
}
Before running the test case add the test-case-specific channel to the map:
getChans[tc.chkey] = make(chan struct{})
Then use the channel key field in the test case as part of the host's URL path:
err := SomeFeature(server.URL+"/"+tc.chkey, tc.variable)
if err != nil {
t.Error("SomeFeature should not error")
}
And to check whether or not http.Get was called use select with some acceptable timeout:
select {
case <-getChans[tc.chkey]:
if !tc.shouldGet {
t.Error(tc.name + " get called")
}
case <-time.Tick(3 * time.Second):
if tc.shouldGet {
t.Error(tc.name + " get not called")
}
}
https://go.dev/play/p/7By3ArkbI_o

Related

Go mockgen, mocked function not called

I am using Go 1.19 on a windows machine with 8 cores, operating system is Windows 10 Pro.
I used the mockgen tool to generate the mock. When I debug my test I see the mocked method is recorded when I execute the EXPECT() function.
The mocked function is called, but the test fails with 'missing call' on the mocked function.
I cannot see what I am doing wrong, can anyone please point it out ?
Directory Structure :
cmd
configure.go
configure_test.go
mocks
mock_validator.go
validator
validator.go
user
user.go
go.mod
main.go
* Contents of main.go
package main
import (
"localdev/mockexample/cmd"
)
func main() {
cmd.Configure()
}
* Contents of configure.go
package cmd
import (
"fmt"
"localdev/mockexample/user"
"os"
"localdev/mockexample/validator"
)
var (
name, password string
)
func Configure() {
name := os.Args[1]
password := os.Args[2]
user, err := validate(validator.NewValidator(name, password))
if err != nil {
fmt.Printf("%v\n", err)
return
}
fmt.Printf("Credentials are valid. Welcome: %s %s\n", user.FirstName, user.LastName)
}
func validate(validator validator.Validator) (*user.Data, error) {
user, err := validator.ValidateUser()
if err != nil {
return nil, fmt.Errorf("some thing went wrong. %v", err)
}
return user, nil
}
* Contents of validator.go
package validator
import (
"fmt"
"localdev/mockexample/user"
)
//go:generate mockgen -destination=../mocks/mock_validator.go -package=mocks localdev/mockexample/validator Validator
type Validator interface {
ValidateUser() (*user.Data, error)
}
type ValidationRequest struct {
Command string
Name string
Password string
}
func (vr ValidationRequest) ValidateUser() (*user.Data, error) {
if vr.Name == "bob" && vr.Password == "1234" {
return &user.Data{UserID: "123", UserName: "bsmith", FirstName: "Bob", LastName: "Smith"}, nil
}
return nil, fmt.Errorf("invalid credentials")
}
func NewValidator(name string, password string) Validator {
return &ValidationRequest{Name: name, Password: password}
}
* Contents of user.go
package user
type Data struct {
UserID string `json:"user_id"`
UserName string `json:"user_name"`
FirstName string `json:"first_name"`
LastName string `json:"last_name"`
}
* Contents of configure_test.go
package cmd
import (
"localdev/mockexample/mocks"
"localdev/mockexample/user"
"os"
"testing"
"github.com/golang/mock/gomock"
)
func TestConfigure(t *testing.T) {
t.Run("ConfigureWithMock", func(t *testing.T) {
os.Args[1] = "bob"
os.Args[2] = "1234"
ctrl := gomock.NewController(t)
mockValidator := mocks.NewMockValidator(ctrl)
//mockValidator.EXPECT().ValidateUser().AnyTimes() // zero more calls, so this will also pass.
userData := user.Data{UserID: "testId"}
mockValidator.EXPECT().ValidateUser().Return(&userData, nil).Times(1) //(gomock.Any(), gomock.Any()) //(&userData, nil)
Configure()
})
}
Contents of generated mock
// Code generated by MockGen. DO NOT EDIT.
// Source: localdev/mockexample/validator (interfaces: Validator)
// Package mocks is a generated GoMock package.
package mocks
import (
user "localdev/mockexample/user"
reflect "reflect"
gomock "github.com/golang/mock/gomock"
)
// MockValidator is a mock of Validator interface.
type MockValidator struct {
ctrl *gomock.Controller
recorder *MockValidatorMockRecorder
}
// MockValidatorMockRecorder is the mock recorder for MockValidator.
type MockValidatorMockRecorder struct {
mock *MockValidator
}
// NewMockValidator creates a new mock instance.
func NewMockValidator(ctrl *gomock.Controller) *MockValidator {
mock := &MockValidator{ctrl: ctrl}
mock.recorder = &MockValidatorMockRecorder{mock}
return mock
}
// EXPECT returns an object that allows the caller to indicate expected use.
func (m *MockValidator) EXPECT() *MockValidatorMockRecorder {
return m.recorder
}
// ValidateUser mocks base method.
func (m *MockValidator) ValidateUser() (*user.Data, error) {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "ValidateUser")
ret0, _ := ret[0].(*user.Data)
ret1, _ := ret[1].(error)
return ret0, ret1
}
// ValidateUser indicates an expected call of ValidateUser.
func (mr *MockValidatorMockRecorder) ValidateUser() *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "ValidateUser", reflect.TypeOf((*MockValidator)(nil).ValidateUser))
}
The root problem is that the function Configure never uses the mock structure, so you get a missing call(s) to *mocks.MockValidator.ValidateUser() error.
In the file configure_test.go, mockValidator is simply not used at all. There must be some kind of injection of that mock in order to be called by the Configure function.
You could make the following changes to fix the test, as an example of what I'm referring to injection. Not saying this is the best approach but I'm trying to make the fewer possible changes to your code.
configure_test.go:
func TestConfigure(t *testing.T) {
t.Run("ConfigureWithMock", func(t *testing.T) {
os.Args[1] = "bob"
os.Args[2] = "1234"
ctrl := gomock.NewController(t)
mockValidator := mocks.NewMockValidator(ctrl)
//mockValidator.EXPECT().ValidateUser().AnyTimes() // zero more calls, so this will also pass.
userData := user.Data{UserID: "testId"}
mockValidator.
EXPECT().
ValidateUser("bob", "1234").
Return(&userData, nil).
Times(1) //(gomock.Any(), gomock.Any()) //(&userData, nil)
Configure(mockValidator)
})
}
configure.go
func Configure(v validator.Validator) {
name := os.Args[1]
password := os.Args[2]
user, err := v.ValidateUser(name, password)
if err != nil {
fmt.Printf("some thing went wrong. %v\n", err)
return
}
fmt.Printf("Credentials are valid. Welcome: %s %s\n", user.FirstName, user.LastName)
}
validator.go
type Validator interface {
ValidateUser(name, password string) (*user.Data, error)
}
type ValidationRequest struct {
Command string
// Name string
// Password string
}
func (vr ValidationRequest) ValidateUser(name, password string) (*user.Data, error) {
if name == "bob" && password == "1234" {
return &user.Data{UserID: "123", UserName: "bsmith", FirstName: "Bob", LastName: "Smith"}, nil
}
return nil, fmt.Errorf("invalid credentials")
}
func NewValidator() Validator {
return &ValidationRequest{}
}
Take into account that you need to generate the mock again. Hope this helps you to understand mock testing.

gomock: because: there are no expected calls of the method "Pod" for that receiver

I am trying to mock mysql, but occur error: "because: there are no expected calls of the method "Pod" for that receiver. "
I confirmed that I have generated the Pod method with the Mockgen tool,
Below is my code
func TestPodService_Create(t *testing.T) {
ctrl := gomock.NewController(t)
defer ctrl.Finish()
mockFactory := store.NewMockFactory(ctrl)
mockPod := store.NewMockPodStore(ctrl)
pods := fake.FakePod(10)
mockPod.EXPECT().Create(gomock.Eq(context.TODO()), gomock.Eq(pods[0])).Return(nil)
type fields struct {
store store.Factory
redisCli redis.RedisCli
}
type args struct {
ctx context.Context
pod *model.Pod
}
tests := []struct {
name string
fields fields
args args
wantErr bool
}{
// TODO: Add test cases.
{
name: "test case 1",
fields: fields{store: mockFactory,},
args: args{
ctx: context.TODO(),
pod: &pods[0],
},
wantErr: false,
},
}
for _, tt := range tests {
fmt.Printf("begin to test\n")
podService := &PodService{store: tt.fields.store}
err := podService.Create(tt.args.ctx, tt.args.pod)
assert.Equal(t, tt.wantErr, err!=nil)
}
}
You need to include this line in your TestPodService_Create():
mockPod.EXPECT().Pod(gomock.Any()).AnyTimes()
Adjust the gomock.Any() and .AnyTimes() for your desired goals.

Why is the response body empty when running a test of mux API?

I am trying to build and test a very basic API in Go to learn more about the language after following their tutorial. The API and the four routes defined work in Postman and the browser, but when trying to write the test for any of the routes, the ResponseRecorder doesn't have a body, so I cannot verify it is correct.
I followed the example here and it works, but when I change it for my route, there is no response.
Here is my main.go file.
package main
import (
"encoding/json"
"fmt"
"log"
"net/http"
"github.com/gorilla/mux"
)
// A Person represents a user.
type Person struct {
ID string `json:"id,omitempty"`
Firstname string `json:"firstname,omitempty"`
Lastname string `json:"lastname,omitempty"`
Location *Location `json:"location,omitempty"`
}
// A Location represents a Person's location.
type Location struct {
City string `json:"city,omitempty"`
Country string `json:"country,omitempty"`
}
var people []Person
// GetPersonEndpoint returns an individual from the database.
func GetPersonEndpoint(w http.ResponseWriter, req *http.Request) {
w.Header().Set("Content-Type", "application/json")
params := mux.Vars(req)
for _, item := range people {
if item.ID == params["id"] {
json.NewEncoder(w).Encode(item)
return
}
}
json.NewEncoder(w).Encode(&Person{})
}
// GetPeopleEndpoint returns all people from the database.
func GetPeopleEndpoint(w http.ResponseWriter, req *http.Request) {
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(people)
}
// CreatePersonEndpoint creates a new person in the database.
func CreatePersonEndpoint(w http.ResponseWriter, req *http.Request) {
w.Header().Set("Content-Type", "application/json")
params := mux.Vars(req)
var person Person
_ = json.NewDecoder(req.Body).Decode(&person)
person.ID = params["id"]
people = append(people, person)
json.NewEncoder(w).Encode(people)
}
// DeletePersonEndpoint deletes a person from the database.
func DeletePersonEndpoint(w http.ResponseWriter, req *http.Request) {
w.Header().Set("Content-Type", "application/json")
params := mux.Vars(req)
for index, item := range people {
if item.ID == params["id"] {
people = append(people[:index], people[index+1:]...)
break
}
}
json.NewEncoder(w).Encode(people)
}
// SeedData is just for this example and mimics a database in the 'people' variable.
func SeedData() {
people = append(people, Person{ID: "1", Firstname: "John", Lastname: "Smith", Location: &Location{City: "London", Country: "United Kingdom"}})
people = append(people, Person{ID: "2", Firstname: "John", Lastname: "Doe", Location: &Location{City: "New York", Country: "United States Of America"}})
}
func main() {
router := mux.NewRouter()
SeedData()
router.HandleFunc("/people", GetPeopleEndpoint).Methods("GET")
router.HandleFunc("/people/{id}", GetPersonEndpoint).Methods("GET")
router.HandleFunc("/people/{id}", CreatePersonEndpoint).Methods("POST")
router.HandleFunc("/people/{id}", DeletePersonEndpoint).Methods("DELETE")
fmt.Println("Listening on http://localhost:12345")
fmt.Println("Press 'CTRL + C' to stop server.")
log.Fatal(http.ListenAndServe(":12345", router))
}
Here is my main_test.go file.
package main
import (
"fmt"
"net/http"
"net/http/httptest"
"testing"
)
func TestGetPeopleEndpoint(t *testing.T) {
req, err := http.NewRequest("GET", "/people", nil)
if err != nil {
t.Fatal(err)
}
// We create a ResponseRecorder (which satisfies http.ResponseWriter) to record the response.
rr := httptest.NewRecorder()
handler := http.HandlerFunc(GetPeopleEndpoint)
// Our handlers satisfy http.Handler, so we can call their ServeHTTP method
// directly and pass in our Request and ResponseRecorder.
handler.ServeHTTP(rr, req)
// Trying to see here what is in the response.
fmt.Println(rr)
fmt.Println(rr.Body.String())
// Check the status code is what we expect.
if status := rr.Code; status != http.StatusOK {
t.Errorf("handler returned wrong status code: got %v want %v",
status, http.StatusOK)
}
// Check the response body is what we expect - Commented out because it will fail because there is no body at the moment.
// expected := `[{"id":"1","firstname":"John","lastname":"Smith","location":{"city":"London","country":"United Kingdom"}},{"id":"2","firstname":"John","lastname":"Doe","location":{"city":"New York","country":"United States Of America"}}]`
// if rr.Body.String() != expected {
// t.Errorf("handler returned unexpected body: got %v want %v",
// rr.Body.String(), expected)
// }
}
I appreciate that I am probably making a beginner mistake, so please take mercy on me. I have read a number of blogs testing mux, but can't see what I have done wrong.
Thanks in advance for your guidance.
UPDATE
Moving my SeeData call to init() resolved the body being empty for the people call.
func init() {
SeedData()
}
However, I now have no body returned when testing a specific id.
func TestGetPersonEndpoint(t *testing.T) {
id := 1
path := fmt.Sprintf("/people/%v", id)
req, err := http.NewRequest("GET", path, nil)
if err != nil {
t.Fatal(err)
}
// We create a ResponseRecorder (which satisfies http.ResponseWriter) to record the response.
rr := httptest.NewRecorder()
handler := http.HandlerFunc(GetPersonEndpoint)
// Our handlers satisfy http.Handler, so we can call their ServeHTTP method
// directly and pass in our Request and ResponseRecorder.
handler.ServeHTTP(rr, req)
// Check request is made correctly and responses.
fmt.Println(path)
fmt.Println(rr)
fmt.Println(req)
fmt.Println(handler)
// expected response for id 1.
expected := `{"id":"1","firstname":"John","lastname":"Smith","location":{"city":"London","country":"United Kingdom"}}` + "\n"
if status := rr.Code; status != http.StatusOK {
message := fmt.Sprintf("The test returned the wrong status code: got %v, but expected %v", status, http.StatusOK)
t.Fatal(message)
}
if rr.Body.String() != expected {
message := fmt.Sprintf("The test returned the wrong data:\nFound: %v\nExpected: %v", rr.Body.String(), expected)
t.Fatal(message)
}
}
Moving my SeedData call to init() resolved the body being empty for the people call.
func init() {
SeedData()
}
Creating a new router instance resolved the issue with accessing a variable on a route.
rr := httptest.NewRecorder()
router := mux.NewRouter()
router.HandleFunc("/people/{id}", GetPersonEndpoint)
router.ServeHTTP(rr, req)
I think its because your test isn't including the router hence the path variables aren't being detected. Here, try this
// main.go
func router() *mux.Router {
router := mux.NewRouter()
router.HandleFunc("/people", GetPeopleEndpoint).Methods("GET")
router.HandleFunc("/people/{id}", GetPersonEndpoint).Methods("GET")
router.HandleFunc("/people/{id}", CreatePersonEndpoint).Methods("POST")
router.HandleFunc("/people/{id}", DeletePersonEndpoint).Methods("DELETE")
return router
}
and in your testcase, initiatize from the router method like below
handler := router()
// Our handlers satisfy http.Handler, so we can call their ServeHTTP method
// directly and pass in our Request and ResponseRecorder.
handler.ServeHTTP(rr, req)
And now if you try accessing the path variable id, it should be present in the map retured by mux since mux registered it when you initiatlized the Handler from mux Router instance returned from router()
params := mux.Vars(req)
for index, item := range people {
if item.ID == params["id"] {
people = append(people[:index], people[index+1:]...)
break
}
}
Also like you mentioned, use the init function for one time setups.
// main.go
func init(){
SeedData()
}

Go converting ptypes/struct Value to BSON

Requirements
Two services:
Server - for writing blog posts to MongoDB
Client - sends request to the first service
The blog post has title of type string, and content which is a dynamic type - can be any JSON value.
Protobuf
syntax = "proto3";
package blog;
option go_package = "blogpb";
import "google/protobuf/struct.proto";
message Blog {
string id = 1;
string title = 2;
google.protobuf.Value content = 3;
}
message CreateBlogRequest {
Blog blog = 1;
}
message CreateBlogResponse {
Blog blog = 1;
}
service BlogService {
rpc CreateBlog (CreateBlogRequest) returns (CreateBlogResponse);
}
Let's start with protobuf message, which meats requirements - string for title and any JSON value for content.
Client
package main
import (...)
func main() {
cc, _ := grpc.Dial("localhost:50051", grpc.WithInsecure())
defer cc.Close()
c := blogpb.NewBlogServiceClient(cc)
var blog blogpb.Blog
json.Unmarshal([]byte(`{"title": "First example", "content": "string"}`), &blog)
c.CreateBlog(context.Background(), &blogpb.CreateBlogRequest{Blog: &blog})
json.Unmarshal([]byte(`{"title": "Second example", "content": {"foo": "bar"}}`), &blog)
c.CreateBlog(context.Background(), &blogpb.CreateBlogRequest{Blog: &blog})
}
The client sends two requests to the server - one with content having string type, and other with the object. No errors here.
Server
package main
import (...)
var collection *mongo.Collection
type server struct {
}
type blogItem struct {
ID primitive.ObjectID `bson:"_id,omitempty"`
Title string `bson:"title"`
Content *_struct.Value `bson:"content"`
}
func (*server) CreateBlog(ctx context.Context, req *blogpb.CreateBlogRequest) (*blogpb.CreateBlogResponse, error) {
blog := req.GetBlog()
data := blogItem{
Title: blog.GetTitle(),
Content: blog.GetContent(),
}
// TODO: convert "data" or "data.Content" to something that could be BSON encoded..
res, err := collection.InsertOne(context.Background(), data)
if err != nil {
log.Fatal(err)
}
oid, _ := res.InsertedID.(primitive.ObjectID)
return &blogpb.CreateBlogResponse{
Blog: &blogpb.Blog{
Id: oid.Hex(),
Title: blog.GetTitle(),
Content: blog.GetContent(),
},
}, nil
}
func main() {
client, _ := mongo.NewClient(options.Client().ApplyURI("mongodb://localhost:27017"))
client.Connect(context.TODO())
collection = client.Database("mydb").Collection("blog")
lis, _ := net.Listen("tcp", "0.0.0.0:50051")
s := grpc.NewServer([]grpc.ServerOption{}...)
blogpb.RegisterBlogServiceServer(s, &server{})
reflection.Register(s)
go func() { s.Serve(lis) }()
ch := make(chan os.Signal, 1)
signal.Notify(ch, os.Interrupt)
<-ch
client.Disconnect(context.TODO())
lis.Close()
s.Stop()
}
And here I get:
cannot transform type main.blogItem to a BSON Document: no encoder
found for structpb.isValue_Kind
What do I expect? To see the exact value of content in MongoDB, something like this:
{ "_id" : ObjectId("5e5f6f75d2679d058eb9ef79"), "title" : "Second example", "content": "string" }
{ "_id" : ObjectId("5e5f6f75d2679d058eb9ef78"), "title" : "First example", "content": { "foo": "bar" } }
I guess I need to transform data.Content in the line where I added TODO:...
I can create github repo with this example if that could help.
So as suggested by #nguyenhoai890 in the comment I managed to fix it using jsonpb lib - first MarshalToString to covert from structpb to string(json) and then json.Unmarshal to convert from string(json) to interface{} which is supported by BSON. Also I had to fix a Client to correctly unmarshal from string to protobuf.
Client
func main() {
cc, _ := grpc.Dial("localhost:50051", grpc.WithInsecure())
defer cc.Close()
c := blogpb.NewBlogServiceClient(cc)
var blog blogpb.Blog
jsonpb.UnmarshalString(`{"title": "Second example", "content": {"foo": "bar"}}`, &blog)
c.CreateBlog(context.Background(), &blogpb.CreateBlogRequest{Blog: &blog})
jsonpb.UnmarshalString(`{"title": "Second example", "content": "stirngas"}`, &blog)
c.CreateBlog(context.Background(), &blogpb.CreateBlogRequest{Blog: &blog})
}
Server
type blogItem struct {
ID primitive.ObjectID `bson:"_id,omitempty"`
Title string `bson:"title"`
Content interface{} `bson:"content"`
}
func (*server) CreateBlog(ctx context.Context, req *blogpb.CreateBlogRequest) (*blogpb.CreateBlogResponse, error) {
blog := req.GetBlog()
contentString, err := new(jsonpb.Marshaler).MarshalToString(blog.GetContent())
if err != nil {
log.Fatal(err)
}
var contentInterface interface{}
json.Unmarshal([]byte(contentString), &contentInterface)
data := blogItem{
Title: blog.GetTitle(),
Content: contentInterface,
}
res, err := collection.InsertOne(context.Background(), data)
if err != nil {
log.Fatal(err)
}
oid, _ := res.InsertedID.(primitive.ObjectID)
return &blogpb.CreateBlogResponse{
Blog: &blogpb.Blog{
Id: oid.Hex(),
Title: blog.GetTitle(),
Content: blog.GetContent(),
},
}, nil
}

Getting same file downloaded multiple times concurrently

I am concurrently downloading files (with a WaitGroup) from a slice of config objects (where each config object contains the URL that needs to be downloaded), but when I use concurrency, I get the same exact data written with every execution.
I believe I included everything below for a minimal reproducible example.
Here are my imports:
package main
import (
"encoding/json"
"fmt"
"io"
"io/ioutil"
"log"
"net/http"
"os"
"path"
"path/filepath"
"strconv"
"strings"
"sync"
)
The method that's looping through my objects and executing the go routine to download each file is here:
func downloadAllFiles(configs []Config) {
var wg sync.WaitGroup
for i, config := range configs {
wg.Add(1)
go config.downloadFile(&wg)
}
wg.Wait()
}
Basically, my function is downloading a file from a URL into a directory stored on NFS.
Here is the download function:
func (config *Config) downloadFile(wg *sync.WaitGroup) {
resp, _ := http.Get(config.ArtifactPathOrUrl)
fmt.Println("Downloading file: " + config.ArtifactPathOrUrl)
fmt.Println(" to location: " + config.getNfsFullFileSystemPath())
defer resp.Body.Close()
nfsDirectoryPath := config.getBaseNFSFileSystemPath()
os.MkdirAll(nfsDirectoryPath, os.ModePerm)
fullFilePath := config.getNfsFullFileSystemPath()
out, err := os.Create(fullFilePath)
if err != nil {
panic(err)
}
defer out.Close()
io.Copy(out, resp.Body)
wg.Done()
}
Here's a minimal part of the Config struct:
type Config struct {
Namespace string `json:"namespace,omitempty"`
Tenant string `json:"tenant,omitempty"`
Name string `json:"name,omitempty"`
ArtifactPathOrUrl string `json:"artifactPathOrUrl,omitempty"`
}
Here are the instance/helper functions:
func (config *Config) getDefaultNfsURLBase() string {
return "http://example.domain.nfs.location.com/"
}
func (config *Config) getDefaultNfsFilesystemBase() string {
return "/data/nfs/location/"
}
func (config *Config) getBaseNFSFileSystemPath() string {
basePath := filepath.Dir(config.getNfsFullFileSystemPath())
return basePath
}
func (config *Config) getNfsFullFileSystemPath() string {
// basePath is like: /data/nfs/location/
trimmedBasePath := strings.TrimSuffix(config.getDefaultNfsFilesystemBase(), "/")
fileName := config.getBaseFileName()
return trimmedBasePath + "/" + config.Tenant + "/" + config.Namespace + "/" + config.Name + "/" + fileName
}
Here is how I'm getting the configs and unmarshalling them:
func getConfigs() string {
b, err := ioutil.ReadFile("pulsarDeploy_example.json")
if err != nil {
fmt.Print(err)
}
str := string(b) // convert content to a 'string'
return str
}
func deserializeJSON(configJson string) []Config {
jsonAsBytes := []byte(configJson)
configs := make([]Config, 0)
err := json.Unmarshal(jsonAsBytes, &configs)
if err != nil {
panic(err)
}
return configs
}
For a minimal example, I think this data for the pulsarDeploy_example.json file should work:
[{ "artifactPathOrUrl": "http://www.java2s.com/Code/JarDownload/sample/sample.jar.zip",
"namespace": "exampleNamespace1",
"name": "exampleName1",
"tenant": "exampleTenant1"
},
{
"artifactPathOrUrl": "http://www.java2s.com/Code/JarDownload/sample-calculator/sample-calculator-bundle-2.0.jar.zip",
"namespace": "exampleNamespace1",
"name": "exampleName2",
"tenant": "exampleTenant1"
},
{
"artifactPathOrUrl": "http://www.java2s.com/Code/JarDownload/helloworld/helloworld.jar.zip",
"namespace": "exampleNamespace1",
"name": "exampleName3",
"tenant": "exampleTenant1"
},
{
"artifactPathOrUrl": "http://www.java2s.com/Code/JarDownload/fabric-activemq/fabric-activemq-demo-7.0.2.fuse-097.jar.zip",
"namespace": "exampleNamespace1",
"name": "exampleName4",
"tenant": "exampleTenant1"
}
]
(Note that the example file URLs were just random Jars I grabbed online.)
When I run my code, instead of downloading each file, it repeatedly downloads the same file, and the information it prints to the console (from the Downloading file: and to location: lines) is exactly the same for each object (instead of printing the messages that are unique to each object), which definitely is a concurrency issue.
This issue reminds me of what happens if you try to run a for loop with a closure and end up locking a single object instance into your loop and executing repeatedly on the same object.
What is causing this behavior, and how do I resolve it?
I'm pretty sure that your guess
This issue reminds me of what happens if you try to run a for loop with a closure and end up locking a single object instance into your loop and executing repeatedly on the same object.
is correct. The simple fix is to "assign to local var" like
for _, config := range configs {
wg.Add(1)
cur := config
go cur.downloadFile(&wg)
}
but I don't like APIs which take waitgroup as a parameter so I suggest
for _, config := range configs {
wg.Add(1)
go func(cur Config) {
defer wg.Done()
cur.downloadFile()
}(config)
}
and change downloadFile signature to func (config *Config) downloadFile() and drop the wg usage in it.

Resources