I want to call two endpoints at the same time (A and B). But if I got a response 200 from both I need to use the response from A otherwise use B response.
If B returns first I need to wait for A, in other words, I must use A whenever A returns 200.
Can you guys help me with the pattern?
Thank you
Wait for a result from A. If the result is not good, then wait from a result from B. Use a buffered channel for the B result so that the sender does not block when A is good.
In the following snippet, fnA() and fnB() functions that issue requests to the endpoints, consume the response and cleanup. I assume that the result is a []byte, but it could be the result of decoding JSON or something else. Here's an example for fnA:
func fnA() ([]byte, error) {
r, err := http.Get("http://example.com/a")
if err != nil {
return nil, err
}
defer r.Body.Close() // <-- Important: close the response body!
if r.StatusCode != 200 {
return nil, errors.New("bad response")
}
return ioutil.ReadAll(r.Body)
}
Define a type to hold the result and error.
type response struct {
result []byte
err error
}
With those preliminaries done, here's how to prioritize A over B.
a := make(chan response)
go func() {
result, err := fnA()
a <- response{result, err}
}()
b := make(chan response, 1) // Size > 0 is important!
go func() {
result, err := fnB()
b <- response{result, err}
}()
resp := <-a
if resp.err != nil {
resp = <-b
if resp.err != nil {
// handle error. A and B both failed.
}
}
result := resp.result
If the application does not execute code concurrently with A and B, then there's no need to use a goroutine for A:
b := make(chan response, 1) // Size > 0 is important!
go func() {
result, err := fnB()
b <- response{result, err}
}()
result, err := fnA()
if err != nil {
resp = <-b
if resp.err != nil {
// handle error. A and B both failed.
}
result = resp.result
}
I'm suggesting you to use something like this, this is a bulky solution, but there you can start more than two endpoints for you needs.
func endpointPriorityTest() {
const (
sourceA = "a"
sourceB = "b"
sourceC = "c"
)
type endpointResponse struct {
source string
response *http.Response
error
}
epResponseChan := make(chan *endpointResponse)
endpointsMap := map[string]string{
sourceA: "https://jsonplaceholder.typicode.com/posts/1",
sourceB: "https://jsonplaceholder.typicode.com/posts/10",
sourceC: "https://jsonplaceholder.typicode.com/posts/100",
}
for source, endpointURL := range endpointsMap {
source := source
endpointURL := endpointURL
go func(respChan chan<- *endpointResponse) {
// You can add a delay so that the response from A takes longer than from B
// and look to the result map
// if source == sourceA {
// time.Sleep(time.Second)
// }
resp, err := http.Get(endpointURL)
respChan <- &endpointResponse{
source: source,
response: resp,
error: err,
}
}(epResponseChan)
}
respCache := make(map[string]*http.Response)
// Reading endpointURL responses from chan
for epResp := range epResponseChan {
// Skips failed requests
if epResp.error != nil {
continue
}
// Save successful response to cache map
respCache[epResp.source] = epResp.response
// Interrupt reading channel if we've got an response from source A
if epResp.source == sourceA {
break
}
}
fmt.Println("result map: ", respCache)
// Now we can use data from cache map
// resp, ok :=respCache[sourceA]
// if ok{
// ...
// }
}
#Zombo 's answer has the correct logic flow. Piggybacking off this, I would suggest one addition: leveraging the context package.
Basically, any potentially blocking tasks should use context.Context to allow the call-chain to perform more efficient clean-up in the event of early cancelation.
context.Context also can be leveraged, in your case, to abort the B call early if the A call succeeds:
func failoverResult(ctx context.Context) *http.Response {
// wrap the (parent) context
ctx, cancel := context.WithCancel(ctx)
// if we return early i.e. if `fnA()` completes first
// this will "cancel" `fnB()`'s request.
defer cancel()
b := make(chan *http.Response, 1)
go func() {
b <- fnB(ctx)
}()
resp := fnA(ctx)
if resp.StatusCode != 200 {
resp = <-b
}
return resp
}
fnA (and fnB) would look something like this:
func fnA(ctx context.Context) (resp *http.Response) {
req, _ := http.NewRequestWithContext(ctx, "GET", aUrl)
resp, _ = http.DefaultClient.Do(req) // TODO: check errors
return
}
Normally in golang, channel are used for communicating between goroutines.
You can orchestrate your scenario with following sample code.
basically you pass channel into your callB which will hold response. You don't need to run callA in goroutine as you always need result from that endpoint/service
package main
import (
"fmt"
"time"
)
func main() {
resB := make(chan int)
go callB(resB)
res := callA()
if res == 200 {
fmt.Print("No Need for B")
} else {
res = <-resB
fmt.Printf("Response from B : %d", res)
}
}
func callA() int {
time.Sleep(1000)
return 200
}
func callB(res chan int) {
time.Sleep(500)
res <- 200
}
Update: As suggestion given in comment, above code leaks "callB"
package main
import (
"fmt"
"time"
)
func main() {
resB := make(chan int, 1)
go callB(resB)
res := callA()
if res == 200 {
fmt.Print("No Need for B")
} else {
res = <-resB
fmt.Printf("Response from B : %d", res)
}
}
func callA() int {
time.Sleep(1000 * time.Millisecond)
return 200
}
func callB(res chan int) {
time.Sleep(500 * time.Millisecond)
res <- 200
}
Related
I have a small Go program that makes a number of requests every tick (1 second). I'm attempting to make these requests concurrently. I want to count and log the number of successful requests made in one tick, and then move on. If requests don't complete in time, I don't want to block the main ticker.
The code below achieves this, but I don't believe I'm closing the channel in concurrentReqs correctly. As any requests that miss the deadline still log with the previous tick. I also believe the ticker in the main function will block waiting for the concurrentReqs to finish. I tried moving the close(ch) inside of the timeout case in my select, but this results in a 'send on closed channel' error.
My understanding is that using contexts with a deadline (probably set in my main ticker) might be a solution for this, but I'm struggling to wrap my head around them, and I wonder if there's something else I can try.
Note: the timeout in concurrentReqs is deliberately low, since I'm testing locally.
package main
import (
"fmt"
"time"
"net/http"
)
type response struct {
num int
statusCode int
requestDuration time.Duration
}
func singleRequest(url string, i int, tick int) response {
start := time.Now()
client := http.Client{ Timeout: 100 * time.Millisecond }
resp, _ := client.Get(url)
fmt.Printf("%d: %d\n", tick, i)
defer resp.Body.Close()
return response{statusCode: int(resp.StatusCode), requestDuration: time.Since(start)}
}
func concurrentReqs(url string, reqsPerTick int, tick int) (results []response){
ch := make(chan response, reqsPerTick)
timeout := time.After(20 * time.Millisecond) // deliberately low
results = make([]response, 0)
for i := 0; i < reqsPerTick; i++ {
go func(i int, t int) {
ch <- singleRequest(url, i, tick)
}(i, tick)
}
for i := 0; i < reqsPerTick; i++ {
select {
case response := <- ch:
results = append(results, response)
case <- timeout:
return
}
}
close(ch)
return
}
func main() {
var url string = "http://end-point.svc/req"
c := time.Tick(1 * time.Second)
for next := range c {
things := concurrentReqs(url, 100, next.Second())
fmt.Printf("%s: Successful Reqs - %d\n", int(next.Second()), len(things))
}
}
I suggest to use context with timeout for cancellation and timing out. Also I think using wait group and mutex protected result writing helps simplicity here by eliminating second loop.
package main
import (
"context"
"fmt"
"log"
"net/http"
"sync"
"time"
)
type response struct {
num int
statusCode int
requestDuration time.Duration
}
func singleRequest(ctx context.Context, url string, i int, tick int) (response, error) {
start := time.Now()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return response{requestDuration: time.Since(start)}, err
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return response{requestDuration: time.Since(start)}, err
}
fmt.Printf("%d: %d\n", tick, i)
defer resp.Body.Close()
return response{statusCode: int(resp.StatusCode), requestDuration: time.Since(start)}, nil
}
func concurrentReqs(url string, reqsPerTick int, tick int) (results []response) {
mu := sync.Mutex{}
results = make([]response, 0)
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
defer cancel()
wg := sync.WaitGroup{}
for i := 0; i < reqsPerTick; i++ {
wg.Add(1)
go func(i int, t int) {
defer wg.Done()
response, err := singleRequest(ctx, url, i, tick)
if err != nil {
log.Print(err)
return
}
mu.Lock()
results = append(results, response)
mu.Unlock()
}(i, tick)
}
wg.Wait()
return results
}
func main() {
var url string = "http://end-point.svc/req"
c := time.Tick(1 * time.Second)
for next := range c {
// You may want to wrap this in a goroutine to make sure tick is not skipped.
// Otherwise if concurrentReqs takes more than a tick time for whatever reason, a tick will be skipped.
things := concurrentReqs(url, 100, next.Second())
fmt.Printf("%s: Successful Reqs - %d\n", int(next.Second()), len(things))
}
}
I try to build concurrent crawler based on Tour and some others SO answers regarding that. What I have currently is below but I think I have here two subtle issues.
Sometimes I get 16 urls in response and sometimes 17 (debug print in main). I know it because when I even change WriteToSlice to Read then in Read sometimes 'Read: end, counter = ' is never reached and it's always when I get 16 urls.
I have troubles with err channel, I get no messages in this channel, even when I run my main Crawl method with address like www.golang.org so without valid schema error should be send via err channel
Concurrency is really difficult topic, help and advice will be appreciated
package main
import (
"fmt"
"net/http"
"sync"
"golang.org/x/net/html"
)
type urlCache struct {
urls map[string]struct{}
sync.Mutex
}
func (v *urlCache) Set(url string) bool {
v.Lock()
defer v.Unlock()
_, exist := v.urls[url]
v.urls[url] = struct{}{}
return !exist
}
func newURLCache() *urlCache {
return &urlCache{
urls: make(map[string]struct{}),
}
}
type results struct {
data chan string
err chan error
}
func newResults() *results {
return &results{
data: make(chan string, 1),
err: make(chan error, 1),
}
}
func (r *results) close() {
close(r.data)
close(r.err)
}
func (r *results) WriteToSlice(s *[]string) {
for {
select {
case data := <-r.data:
*s = append(*s, data)
case err := <-r.err:
fmt.Println("e ", err)
}
}
}
func (r *results) Read() {
fmt.Println("Read: start")
counter := 0
for c := range r.data {
fmt.Println(c)
counter++
}
fmt.Println("Read: end, counter = ", counter)
}
func crawl(url string, depth int, wg *sync.WaitGroup, cache *urlCache, res *results) {
defer wg.Done()
if depth == 0 || !cache.Set(url) {
return
}
response, err := http.Get(url)
if err != nil {
res.err <- err
return
}
defer response.Body.Close()
node, err := html.Parse(response.Body)
if err != nil {
res.err <- err
return
}
urls := grablUrls(response, node)
res.data <- url
for _, url := range urls {
wg.Add(1)
go crawl(url, depth-1, wg, cache, res)
}
}
func grablUrls(resp *http.Response, node *html.Node) []string {
var f func(*html.Node) []string
var results []string
f = func(n *html.Node) []string {
if n.Type == html.ElementNode && n.Data == "a" {
for _, a := range n.Attr {
if a.Key != "href" {
continue
}
link, err := resp.Request.URL.Parse(a.Val)
if err != nil {
continue
}
results = append(results, link.String())
}
}
for c := n.FirstChild; c != nil; c = c.NextSibling {
f(c)
}
return results
}
res := f(node)
return res
}
// Crawl ...
func Crawl(url string, depth int) []string {
wg := &sync.WaitGroup{}
output := &[]string{}
visited := newURLCache()
results := newResults()
defer results.close()
wg.Add(1)
go crawl(url, depth, wg, visited, results)
go results.WriteToSlice(output)
// go results.Read()
wg.Wait()
return *output
}
func main() {
r := Crawl("https://www.golang.org", 2)
// r := Crawl("www.golang.org", 2) // no schema, error should be generated and send via err
fmt.Println(len(r))
}
Both your questions 1 and 2 are a result of the same bug.
In Crawl() you are not waiting for this go routine to finish: go results.WriteToSlice(output). On the last crawl() function, the wait group is released, the output is returned and printed before the WriteToSlice function finishes with the data and err channel. So what has happened is this:
crawl() finishes, placing data in results.data and results.err.
Waitgroup wait() unblocks, causing main() to print the length of the result []string
WriteToSlice adds the last data (or err) item to the channel
You need to return from Crawl() not only when the data is done being written to the channel, but also when the channel is done being read in it's entirety (including the buffer). A good way to do this is close channels when you are sure that you are done with them. By organizing your code this way, you can block on the go routine that is draining the channels, and instead of using the wait group to release to main, you wait until the channels are 100% done.
You can see this gobyexample https://gobyexample.com/closing-channels. Remember that when you close a channel, the channel can still be used until the last item is taken. So you can close a buffered channel, and the reader will still get all the items that were queued in the channel.
There is some code structure that can change to make this cleaner, but here is a quick way to fix your program. Change Crawl to block on WriteToSlice. Close the data channel when the crawl function finishes, and wait for WriteToSlice to finish.
// Crawl ...
func Crawl(url string, depth int) []string {
wg := &sync.WaitGroup{}
output := &[]string{}
visited := newURLCache()
results := newResults()
go func() {
wg.Add(1)
go crawl(url, depth, wg, visited, results)
wg.Wait()
// All data is written, this makes `WriteToSlice()` unblock
close(results.data)
}()
// This will block until results.data is closed
results.WriteToSlice(output)
close(results.err)
return *output
}
Then on write to slice, you have to check for the closed channel to exit the for loop:
func (r *results) WriteToSlice(s *[]string) {
for {
select {
case data, open := <-r.data:
if !open {
return // All data done
}
*s = append(*s, data)
case err := <-r.err:
fmt.Println("e ", err)
}
}
}
Here is the full code: https://play.golang.org/p/GBpGk-lzrhd (it won't work in the playground)
I am not necessarily trying to accomplish something specific, more just understand how goroutines, channels, waitgroups, and select (on channels) plays together. I am writing a simple program that loops through an slice of URLs, fetches the URL, then basically just ends. The simple idea is that I want all of the fetches to occur and return, send their data over channels, and then end once all fetches have occurred. I am almost there, and I know I am missing something in my select that will end the loop, something to say "hey the waitgroup is empty now", but I am unsure how to best do that. Mind taking a look and clearing it up for me? Right now everything runs just fine, it just doesn't terminate, so clearly I am missing something and/or not understanding how some of these components should work together.
package main
import (
"fmt"
"io/ioutil"
"net/http"
"sync"
)
var urls = []string{
"https://www.google.com1",
"https://www.gentoo.org",
}
var wg sync.WaitGroup
// simple struct to store fetching
type urlObject struct {
url string
success bool
body string
}
func getPage(url string, channelMain chan urlObject, channelError chan error) {
// increment waitgroup, defer decrementing
wg.Add(1)
defer wg.Done()
fmt.Println("fetching " + url)
// create a urlObject
uO := urlObject{
url: url,
success: false,
}
// get URL
response, getError := http.Get(url)
// close response later on
if response != nil {
defer response.Body.Close()
}
// send error over error channel if one occurs
if getError != nil {
channelError <- getError
return
}
// convert body to []byte
body, conversionError := ioutil.ReadAll(response.Body)
// convert []byte to string
bodyString := string(body)
// if a conversion error happens send it over the error channel
if conversionError != nil {
channelError <- conversionError
} else {
// if not send a urlObject over the main channel
uO.success = true
uO.body = bodyString
channelMain <- uO
}
}
func main() {
var channelMain = make(chan urlObject)
var channelError = make(chan error)
for _, v := range urls {
go getPage(v, channelMain, channelError)
}
// wait on goroutines to finish
wg.Wait()
for {
select {
case uO := <-channelMain:
fmt.Println("completed " + uO.url)
case err := <-channelError:
fmt.Println("error: " + err.Error())
}
}
}
You need to make the following changes:
As people have mentioned, you probably want to call wg.Add(1) in the main function, before calling your goroutine. That way you KNOW it occurs before the defer wg.Done() call.
Your channel reads will block, unless you can figure out a way to either close the channels in your goroutines, or make them buffered. Probably the easiest way is to make them buffered, e.g., var channelMain = make(chan urlObject, len(urls))
The break in your select statement is going to only exit the select, not the containing for loop. You can label the for loop and break to that, or use some sort of conditional variable.
Playground link to working version: https://play.golang.org/p/WH1fm2MhP-L
package main
import (
"fmt"
"io/ioutil"
"net/http"
"sync"
)
var urls = []string{
"https://www.google.com1",
"https://www.gentoo.org",
}
var wg sync.WaitGroup
// simple struct to store fetching
type urlObject struct {
url string
success bool
body string
}
func getPage(url string, channelMain chan urlObject, channelError chan error) {
// increment waitgroup, defer decrementing
defer wg.Done()
fmt.Println("fetching " + url)
// create a urlObject
uO := urlObject{
url: url,
success: false,
}
// get URL
response, getError := http.Get(url)
// close response later on
if response != nil {
defer response.Body.Close()
}
// send error over error channel if one occurs
if getError != nil {
channelError <- getError
return
}
// convert body to []byte
body, conversionError := ioutil.ReadAll(response.Body)
// convert []byte to string
bodyString := string(body)
// if a conversion error happens send it over the error channel
if conversionError != nil {
channelError <- conversionError
} else {
// if not send a urlObject over the main channel
uO.success = true
uO.body = bodyString
channelMain <- uO
}
}
func main() {
var channelMain = make(chan urlObject, len(urls))
var channelError = make(chan error, len(urls))
for _, v := range urls {
wg.Add(1)
go getPage(v, channelMain, channelError)
}
// wait on goroutines to finish
wg.Wait()
for done := false; !done; {
select {
case uO := <-channelMain:
fmt.Println("completed " + uO.url)
case err := <-channelError:
fmt.Println("error: " + err.Error())
default:
done = true
}
}
}
In a function definition, if a channel is an argument without a direction, does it have to send or receive something?
func makeRequest(url string, ch chan<- string, results chan<- string) {
start := time.Now()
resp, err := http.Get(url)
defer resp.Body.Close()
if err != nil {
fmt.Printf("%v", err)
}
resp, err = http.Post(url, "text/plain", bytes.NewBuffer([]byte("Hey")))
defer resp.Body.Close()
secs := time.Since(start).Seconds()
if err != nil {
fmt.Printf("%v", err)
}
// Cannot move past this.
ch <- fmt.Sprintf("%f", secs)
results <- <- ch
}
func MakeRequestHelper(url string, ch chan string, results chan string, iterations int) {
for i := 0; i < iterations; i++ {
makeRequest(url, ch, results)
}
for i := 0; i < iterations; i++ {
fmt.Println(<-ch)
}
}
func main() {
args := os.Args[1:]
threadString := args[0]
iterationString := args[1]
url := args[2]
threads, err := strconv.Atoi(threadString)
if err != nil {
fmt.Printf("%v", err)
}
iterations, err := strconv.Atoi(iterationString)
if err != nil {
fmt.Printf("%v", err)
}
channels := make([]chan string, 100)
for i := range channels {
channels[i] = make(chan string)
}
// results aggregate all the things received by channels in all goroutines
results := make(chan string, iterations*threads)
for i := 0; i < threads; i++ {
go MakeRequestHelper(url, channels[i], results, iterations)
}
resultSlice := make([]string, threads*iterations)
for i := 0; i < threads*iterations; i++ {
resultSlice[i] = <-results
}
}
In the above code,
ch <- or <-results
seems to be blocking every goroutine that executes makeRequest.
I am new to concurrency model of Go. I understand that sending to and receiving from a channel blocks but find it difficult what is blocking what in this code.
I'm not really sure that you are doing... It seems really convoluted. I suggest you read up on how to use channels.
https://tour.golang.org/concurrency/2
That being said you have so much going on in your code that it was much easier to just gut it to something a bit simpler. (It can be simplified further). I left comments to understand the code.
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
"sync"
"time"
)
// using structs is a nice way to organize your code
type Worker struct {
wg sync.WaitGroup
semaphore chan struct{}
result chan Result
client http.Client
}
// group returns so that you don't have to send to many channels
type Result struct {
duration float64
results string
}
// closing your channels will stop the for loop in main
func (w *Worker) Close() {
close(w.semaphore)
close(w.result)
}
func (w *Worker) MakeRequest(url string) {
// a semaphore is a simple way to rate limit the amount of goroutines running at any single point of time
// google them, Go uses them often
w.semaphore <- struct{}{}
defer func() {
w.wg.Done()
<-w.semaphore
}()
start := time.Now()
resp, err := w.client.Get(url)
if err != nil {
log.Println("error", err)
return
}
defer resp.Body.Close()
// don't have any examples where I need to also POST anything but the point should be made
// resp, err = http.Post(url, "text/plain", bytes.NewBuffer([]byte("Hey")))
// if err != nil {
// log.Println("error", err)
// return
// }
// defer resp.Body.Close()
secs := time.Since(start).Seconds()
b, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Println("error", err)
return
}
w.result <- Result{duration: secs, results: string(b)}
}
func main() {
urls := []string{"https://facebook.com/", "https://twitter.com/", "https://google.com/", "https://youtube.com/", "https://linkedin.com/", "https://wordpress.org/",
"https://instagram.com/", "https://pinterest.com/", "https://wikipedia.org/", "https://wordpress.com/", "https://blogspot.com/", "https://apple.com/",
}
workerNumber := 5
worker := Worker{
semaphore: make(chan struct{}, workerNumber),
result: make(chan Result),
client: http.Client{Timeout: 5 * time.Second},
}
// use sync groups to allow your code to wait for
// all your goroutines to finish
for _, url := range urls {
worker.wg.Add(1)
go worker.MakeRequest(url)
}
// by declaring wait and close as a seperate goroutine
// I can get to the for loop below and iterate on the results
// in a non blocking fashion
go func() {
worker.wg.Wait()
worker.Close()
}()
// do something with the results channel
for res := range worker.result {
fmt.Printf("Request took %2.f seconds.\nResults: %s\n\n", res.duration, res.results)
}
}
The channels in channels are nil (no make is executed; you make the slice but not the channels), so any send or receive will block. I'm not sure exactly what you're trying to do here, but that's the basic problem.
See https://golang.org/doc/effective_go.html#channels for an explanation of how channels work.
This has been the bane of my existence.
type ec2Params struct {
sess *session.Session
region string
}
type cloudwatchParams struct {
cl cloudwatch.CloudWatch
id string
metric string
region string
}
type request struct {
ec2Params
cloudwatchParams
}
// Control concurrency and sync
var maxRoutines = 128
var sem chan bool
var req chan request
func main() {
sem := make(chan bool, maxRoutines)
for i := 0; i < maxRoutines; i++ {
sem <- true
}
req := make(chan request)
go func() { // This is my the producer
for _, arn := range arns {
arnCreds := startSession(arn)
for _, region := range regions {
sess, err := session.NewSession(
&aws.Config{****})
if err != nil {
failOnError(err, "Can't assume role")
}
req <- request{ec2Params: ec2Params{ **** }}
}
}
}()
for f := range(req) {
<- sem
if (ec2Params{}) != f.ec2Params {
go getEC2Metrics(****)
} else {
// I should be excercising this line of code too,
// but I'm not :(
go getMetricFromCloudwatch(****)
}
sem <- true
}
}
getEC2Metrics and getCloudwatchMetrics are the goroutines to execute
func getMetricFromCloudwatch(cl cloudwatch.CloudWatch, id, metric, region string) {
// Magic
}
func getEC2Metrics(sess *session.Session, region string) {
ec := ec2.New(sess)
var ids []string
l, err := ec.DescribeInstances(&ec2.DescribeInstancesInput{})
if err != nil {
fmt.Println(err.Error())
} else {
for _, rsv := range l.Reservations {
for _, inst := range rsv.Instances {
ids = append(ids, *inst.InstanceId)
}
}
metrics := cfg.AWSMetric.Metric
if len(ids) >= 0 {
cl := cloudwatch.New(sess)
for _, id := range ids{
for _, metric := range metrics {
// For what I can tell, execution get stuck here
req <- request{ cloudwatchParams: ***** }}
}
}
}
}
}
Both the anonymous producer in main and getEC2Metrics should publish data to req asynchronically, but so far it seems like whatever getEC2Metrics is publishing to the channel is never processed.
It looks like there is something stopping me from publishing from within a goroutine, but I haven't found anything. I would love to know how to go about this and to produce the indended behavior (This is, an actually working semaphore).
The base of the implementation can be found here: https://burke.libbey.me/conserving-file-descriptors-in-go/
Im frantic, JimB's comment made the wheel spin and now I've solved this!
// Control concurrency and sync
var maxRoutines = 128
var sem chan bool
var req chan request // Not reachable inside getEC2Metrics
func getEC2Metrics(sess *session.Session, region string, req chan <- request ) {
....
....
for _, id := range ids{
for _, metric := range metrics {
req <- request{ **** }} // When using the global req,
// this would block
}
}
....
....
}
func main() {
sem := make(chan bool, maxRoutines)
for i := 0; i < maxRoutines; i++ {
sem <- true
}
req := make(chan request)
go func() {
// Producing tasks
}()
for f := range(req) {
<- sem // checking out tickets outside the goroutine does block
//outside of the goroutine
go func() {
defer func() { sem <- true }()
if (ec2Params{}) != f.ec2Params {
getEC2Metrics(****, req) // somehow sending the channel makes
// possible to publish to it
} else {
getMetricFromCloudwatch(****)
}
}()
}
}
There were two issues:
The semaphore was not locking (I think it is because I was checking out and in tokens inside a goroutine, so there was a race condition probably).
For some reason, The global channel req was not being addressed properly by getEC2Metrics, so it would leave alll the goroutines stuck while trying to publish to a channel that was apparently on scope, but it wasn't (I really don't know why yet).
I've honestly just had luck with the second item, so far I haven't found any docs regarding this quirk, but at the end I'm glad it's working.