Is there a non-blocking http client? [closed] - go

Question

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 3 years ago.
Improve this question
Many languages have their own high-level non-blocking HTTP client, for example, python's aiohttp. Namely, they send out HTTP requests; do not wait for response; When response arrives they make some kind of callbacks.
My questions are
is there a Go package for that?
or we just create a goroutine in which we use normal HTTP clients?
which way is better?

Answer 1

Other languages have such features because when they block waiting for request they block the thread they are using. This is the case for Java, Python or NodeJS. Therefore to make them useful, the developers needed to implement such long-standing blocking operations with callbacks. The root cause of that is the usage of the C library beneath that blocks threads on input-output operations.
Go does not use C library (only in some cases, but it can be turned off) and makes system calls by itself. While doing this the thread that executes current goroutine parks it and executes another goroutine. Therefore you can have enormous number of blocked goroutines without running out of threads. Goroutines are cheap with regard to memory, threads are operating system entities.
In Go using goroutines is better. There is no need for creating asynchronous client because of the above.
For comparison in Java you would quickly end up with multiple threads. The next step would be pooling them as they are costly. Pooling means limiting the concurrency.

Answer 2

As others have stated, goroutines are the way to go (pun intended).
Minimal Example:
type nonBlocking struct {
Response *http.Response
Error error
}
const numRequests = 2
func main() {
nb := make(chan nonBlocking, numRequests)
wg := &sync.WaitGroup{}
for i := 0; i < numRequests; i++ {
wg.Add(1)
go Request(nb)
}
go HandleResponse(nb, wg)
wg.Wait()
}
func Request(nb chan nonBlocking) {
resp, err := http.Get("http://example.com")
nb <- nonBlocking{
Response: resp,
Error: err,
}
}
func HandleResponse(nb chan nonBlocking, wg *sync.WaitGroup) {
for get := range nb {
if get.Error != nil {
log.Println(get.Error)
} else {
log.Println(get.Response.Status)
}
wg.Done()
}
}

Answer 3

Yip, built into the standard library, just not usable by a simple function call out of the box.
Take this example
package main
import (
"flag"
"log"
"net/http"
"sync"
"time"
)
var url string
var timeout time.Duration
func init() {
flag.StringVar(&url, "url", "http://www.stackoverflow.com", "url to GET")
flag.DurationVar(&timeout, "timeout", 5*time.Second, "timeout for the GET operation")
}
func main() {
flag.Parse()
// We use the channel as our means to
// hand the response over
rc := make(chan *http.Response)
// We need a waitgroup because all goroutines exit when main exits
var wg sync.WaitGroup
// We are spinning up an async request
// Increment the counter for our WaitGroup.
// What we are basically doing here is to tell the WaitGroup
// "Hey, there is one more task you have to wait for!"
wg.Add(1)
go func() {
// Notify the WaitGroup that one task is done as soon
// as we exit the goroutine.
defer wg.Done()
log.Printf("Doing GET request on \"%s\"", url)
resp, err := http.Get(url)
if err != nil {
log.Printf("GET for %s: %s", url, err)
}
// We send the reponse downstream
rc <- resp
// Now, the goroutine exits, the defered call to wg.Done()
// is executed.
}()
// And here we do our async processing.
// Note that you could have done the processing in the first goroutine
// as well, since http.Get would be a blocking operation and any subsequent
// code in the goroutine would have been excuted only after the Get returned.
// However, I put te processing into its own goroutine for demonstration purposes.
wg.Add(1)
go func() {
// As above
defer wg.Done()
log.Println("Doing something else")
// Setting up a timer for a timeout.
// Note that this could be done using a request with a context, as well.
to := time.NewTimer(timeout).C
select {
case <-to:
log.Println("Timeout reached")
// Exiting the goroutine, the deferred call to wg.Done is executed
return
case r := <-rc:
if r == nil {
log.Printf("Got no useful response from GETting \"%s\"", url)
// Exiting the goroutine, the deferred call to wg.Done is executed
return
}
log.Printf("Got response with status code %d (%s)", r.StatusCode, r.Status)
log.Printf("Now I can do something useful with the response")
}
}()
// Now we have set up all of our tasks,
// we are waiting until all of them are done...
wg.Wait()
log.Println("All tasks done, exiting")
}
If you look at this closely, we have all building blocks to make GETting an URL and processing the response async. We can start to abstract this a bit:
package main
import (
"flag"
"log"
"net/http"
"time"
)
var url string
var timeout time.Duration
func init() {
flag.StringVar(&url, "url", "http://www.stackoverflow.com", "url to GET")
flag.DurationVar(&timeout, "timeout", 5*time.Second, "timeout for the GET operation")
}
type callbackFunc func(*http.Response, error) error
func getWithCallBack(u string, callback callbackFunc) chan error {
// We create a channel which we can use to notify the caller of the
// result of the callback.
c := make(chan error)
go func() {
c <- callback(http.Get(u))
}()
return c
}
func main() {
flag.Parse()
c := getWithCallBack(url, func(resp *http.Response, err error) error {
if err != nil {
// Doing something useful with the err.
// Add additional cases as needed.
switch err {
case http.ErrNotSupported:
log.Printf("GET not supported for \"%s\"", url)
}
return err
}
log.Printf("GETting \"%s\": Got response with status code %d (%s)", url, resp.StatusCode, resp.Status)
return nil
})
if err := <-c; err != nil {
log.Printf("Error GETting \"%s\": %s", url, err)
}
log.Println("All tasks done, exiting")
}
And there you Go (pun intended): Async processing of GET requests.