I want to write requests to one file from some ajax script. The problem arises when there will be many of those in a second and writing to file will take more time than the break between requests, and when there will be two requests at the same time.
How could I solve this?
I've came up using mutex, like:
var mu sync.Mutex
func writeToFile() {
mu.Lock()
defer mu.Unlock()
// write to file
}
But it makes the whole thing synchronous and I don't really know what happens when there are two requests at the same time. And it still does not lock the file itself.
Uh, what's the proper way to do this?
You only need to make writing to the file "sequential", meaning don't allow 2 concurrent goroutines to write to the file. Yes, if you use locking in the writeToFile() function, serving your ajax requests may become (partially) sequential too.
What I suggest is open the file once, when your application starts. And designate a single goroutine which will be responsible writing to the file, no other goroutines should do it.
And use a buffered channel to send data that should be written to the file. This will make serving ajax requests non-blocking, and still the file will not be written concurrently / parallel.
Note that this way ajax requests won't even have to wait while the data is actually written to the file (faster response time). This may or may not be a problem. For example if later writing fails, your ajax response might already be committed => no chance to signal failure to the client.
Example how to do it:
var (
f *os.File
datach = make(chan []byte, 100) // Buffered channel
)
func init() {
// Open file for appending (create if doesn't exist)
var err error
f, err = os.OpenFile("data.txt", os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0666)
if err != nil {
panic(err)
}
// Start goroutine which writes to the file
go writeToFile()
}
func writeToFile() {
// Loop through any data that needs to be written:
for data := range datach {
if _, err := f.Write(data); err != nil {
// handle error!
}
}
// We get here if datach is closed: shutdown
f.Close()
}
func ajaxHandler(w http.ResponseWriter, r *http.Request) {
// Assmeble data that needs to be written (appended) to the file
data := []byte{1, 2, 3}
// And send it:
datach <- data
}
To gracefully exit from the app, you should close the datach channel: when it's closed, the loop in the writeToFile() will terminate, and the file will be closed (flushing any cached data, releasing OS resources).
If you want to write text to the file, you may declare the data channel like this:
var datach = make(chan string, 100) // Buffered channel
And you may use File.WriteString() to write it to the file:
if _, err := f.WriteString(data); err != nil {
// handle error!
}
Related
I am trying to construct a receiver and sender pattern using two channels in Golang. I am doing a task (API call), and receiving back a Response struct. My goal is that when a response is received I'd like to send it to another channel (writeChan) for additional processing.
I'd like to continuously read/listen on that receiver channel (respChan) and process anything that comes through (such as a Response). Then I'd like to spin up a thread to go and do a further operation with that Response in another goroutine.
I'd like to understand how I can chain together this pattern to allow data to flow from the API calls and concurrently write it (each Response will be written to a separate file destination which the Write() func handles.
Essentially my current pattern is the following:
package main
import (
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup
respChan := make(chan Response) // Response is a struct that contains API response metadata
defer close(respChan)
// requests is just a slice of requests to be made to an API
// This part is working well
for _, req := range requests {
wg.Add(1)
go func(r Request) {
defer wg.Done()
resp, _ := r.Get() // Make the API call and receive back a Response struct
respChan <- resp // Put the response into our channel
}(req)
}
// Now, I want to extract the responses as they become available and send them to another function to do some processing. This I am unsure of how to handle properly
writeChan := make(chan string)
defer close(writeChan)
select {
case resp := <-respChan: // receive from response channel
go func(response Response) {
signal, _ := Write(response) // Separate func to write the response to a file. Not important here in this context.
writeChan <- signal // Put the signal data into the channel which is a string file path of where the file was written (will be used for a later process)
}(resp)
case <-time.After(15 *time.Second):
fmt.Println("15 seconds have passed without receiving anything...")
}
wg.Wait()
}
Let me share with you a working example that you can benefit from. First, I'm gonna present the code, then, I'm gonna walk you through all the relevant sections.
package main
import (
"fmt"
"net/http"
"os"
"strings"
"time"
)
type Request struct {
Url string
DelayInSeconds time.Duration
}
type Response struct {
Url string
StatusCode int
}
func main() {
requests := []Request{
{"https://www.google.com", 0},
{"https://stackoverflow.com", 1},
{"https://www.wikipedia.com", 4},
}
respChan := make(chan Response)
defer close(respChan)
for _, req := range requests {
go func(r Request) {
fmt.Printf("%q - %v\n", r.Url, strings.Repeat("#", 30))
// simulate heavy work
time.Sleep(time.Second * r.DelayInSeconds)
resp, _ := http.Get(r.Url)
res := Response{r.Url, resp.StatusCode}
fmt.Println(time.Now())
respChan <- res
}(req)
}
writeChan := make(chan struct{})
defer close(writeChan)
for i := 0; i < len(requests); i++ {
select {
case res := <-respChan:
go func(r Response) {
f, err := os.Create(fmt.Sprintf("%v.txt", strings.Replace(r.Url, "https://", "", 1)))
if err != nil {
panic(err)
}
defer f.Close()
f.Write([]byte(fmt.Sprintf("%q OK with %d\n", r.Url, r.StatusCode)))
writeChan <- struct{}{}
}(res)
case <-time.After(time.Second * 2):
fmt.Println("Timeout")
}
}
}
Set up
First, I've defined the two structs that will be used in the example: Request and Response. In the former, I put a DelayInSeconds to mock some heavy loads and time-consuming operations. Then, I defined the requests variable that contains all the requests that have to be done.
The writing part
Here, I range over the requests variable. For each request, I'm gonna issue an HTTP request to the target URL. The time.Sleep emulate the heavy load. Then, I write the response to the respChan channel which is unbuffered.
The reading part
Here, the major change is to wrap the select construct into a for loop. Thanks to this, we'll make sure to iterate the right times (based on the length of the requests variable).
Final notes
First of all, bear in mind that the code is oversimplified just to show off the relevant parts. Due to this, a lot of error handling is missing and some inline functions could be extrapolated into named functions. You don't need to use sync.WaitGroup to achieve what you need, the usage of channels will be enough.
Feel free to play with delays and check which files are written!
Let me know if this helps you!
Edit
As requested, I'm gonna provide you with a more accurate solution based on your needs. The new reading part will be something like the following:
count := 0
for {
// this check is need to exit the for loop and not wait indefinitely
// it can be removed based on your needs
if count == 3 {
fmt.Println("all responses arrived...")
return
}
res := <-respChan
count++
go func(r Response) {
f, err := os.Create(fmt.Sprintf("%v.txt", strings.Replace(r.Url, "https://", "", 1)))
if err != nil {
panic(err)
}
defer f.Close()
f.Write([]byte(fmt.Sprintf("%q OK with %d\n", r.Url, r.StatusCode)))
writeChan <- struct{}{}
}(res)
}
Here, the execution is waiting indefinitely within the for loop. No matter how long each request takes to complete, it will be fetched as soon as it arrives. I put, at the top of the for loop, an if to exit after it processed the requests that we need. However, you can avoid it and let the code run till a cancellation signal comes in (it's up to you).
Let me know if this better meets your requirements, thanks!
I am trying to read multiple files in parallel in such a way so that each go routine that is reading a file write its data to that channel, then have a single go-routine that listens to that channel and adds the data to the map. Here is my play.
Below is the example from the play:
package main
import (
"fmt"
"sync"
)
func main() {
var myFiles = []string{"file1", "file2", "file3"}
var myMap = make(map[string][]byte)
dataChan := make(chan fileData, len(myFiles))
wg := sync.WaitGroup{}
defer close(dataChan)
// we create a wait group of N
wg.Add(len(myFiles))
for _, file := range myFiles {
// we create N go-routines, one per file, each one will return a struct containing their filename and bytes from
// the file via the dataChan channel
go getBytesFromFile(file, dataChan, &wg)
}
// we wait until the wait group is decremented to zero by each instance of getBytesFromFile() calling waitGroup.Done()
wg.Wait()
for i := 0; i < len(myFiles); i++ {
// we can now read from the data channel N times.
file := <-dataChan
myMap[file.name] = file.bytes
}
fmt.Printf("%+v\n", myMap)
}
type fileData struct {
name string
bytes []byte
}
// how to handle error from this method if reading file got messed up?
func getBytesFromFile(file string, dataChan chan fileData, waitGroup *sync.WaitGroup) {
bytes := openFileAndGetBytes(file)
dataChan <- fileData{name: file, bytes: bytes}
waitGroup.Done()
}
func openFileAndGetBytes(file string) []byte {
return []byte(fmt.Sprintf("these are some bytes for file %s", file))
}
Problem Statement
How can I use golang.org/x/sync/errgroup to wait on and handle errors from goroutines or if there is any better way like using semaphore? For example if any one of my go routine fails to read data from file then I want to cancels all those remaining in the case of any one routine returning an error (in which case that error is the one bubble back up to the caller). And it should automatically waits for all the supplied go routines to complete successfully for success case.
I also don't want to spawn 100 go-routines if total number of files is 100. I want to control the parallelism if possible if there is any way.
How can I use golang.org/x/sync/errgroup to wait on and handle errors from goroutines or if there is any better way like using semaphore? For example [...] I want to cancels all those remaining in the case of any one routine returning an error (in which case that error is the one bubble back up to the caller). And it should automatically waits for all the supplied go routines to complete successfully for success case.
There are many ways to communicate error states across goroutines. errgroup does a bunch of heavy lifting though, and is appropriate for this case. Otherwise you're going to end up implementing the same thing.
To use errgroup we'll need to handle errors (and for your demo, generate some). In addition, to cancel existing goroutines, we'll use a context from errgroup.NewWithContext.
From the errgroup reference,
Package errgroup provides synchronization, error propagation, and Context cancelation for groups of goroutines working on subtasks of a common task.
Your play doesn't support any error handling. We can't collect and cancel on errors if we don't do any error handling. So I added some code to inject error handling:
func openFileAndGetBytes(file string) (string, error) {
if file == "file2" {
return "", fmt.Errorf("%s cannot be read", file)
}
return fmt.Sprintf("these are some bytes for file %s", file), nil
}
Then that error had to be passed back from getBytesFromFile as well:
func getBytesFromFile(file string, dataChan chan fileData) error {
bytes, err := openFileAndGetBytes(file)
if err == nil {
dataChan <- fileData{name: file, bytes: bytes}
}
return err
}
Now that we've done that, we can turn our attention to how we're going to start up a number of goroutines.
I also don't want to spawn 100 go-routines if total number of files is 100. I want to control the parallelism if possible if there is any way.
Written well, the number of tasks, channel size, and number of workers are typically independent values. The trick is to use channel closure - and in your case, context cancellation - to communicate state between the goroutines. We'll need an additional channel for the distribution of filenames, and an additional goroutine for the collection of the results.
To illustate this point, my code uses 3 workers, and adds a few more files. My channels are unbuffered. This allows us to see some of the files get processed, while others are aborted. If you buffer the channels, the example will still work, but it's more likely for additional work to be processed before the cancellation is handled. Experiment with buffer size along with worker count and number of files to process.
var myFiles = []string{"file1", "file2", "file3", "file4", "file5", "file6"}
fileChan := make(chan string)
dataChan := make(chan fileData)
To start up the workers, instead of starting one for each file, we start the number we desire - here, 3.
for i := 0; i < 3; i++ {
worker_num := i
g.Go(func() error {
for file := range fileChan {
if err := getBytesFromFile(file, dataChan); err != nil {
fmt.Println("worker", worker_num, "failed to process", file, ":", err.Error())
return err
} else if err := ctx.Err(); err != nil {
fmt.Println("worker", worker_num, "context error in worker:", err.Error())
return err
}
}
fmt.Println("worker", worker_num, "processed all work on channel")
return nil
})
}
The workers call your getBytesFromFile function. If it returns an err, we return an err. errgroup will cancel our context automatically in this case. However, the exact order of operations is not deterministic, so more files may or may not get processed before the context is cancelled. I'll show several possibilties below.
by rangeing over fileChan, the worker automatically picks up end of work from the channel closure. If we get an error, we can return it to errgroup immediately. Otherwise, if the context has been cancelled, we can return the cancellation error immediately.
You might think that g.Go would automatically cancel our function. But it cannot. There is no way to cancel a running function in Go other than process termination. errgroup.Group.Go's function argument must cancel itself when appropriate based on the state of its context.
Now we can turn our attention to the thing that puts the files on fileChan. We have 2 options here: we can use a buffered channel of the size of myFiles, like you did. We can fill the entire channel with pending jobs. This is only an option if you know the number of jobs when you create the channel. The other option is to use an additional "distribution" goroutine that can block on writes to fileChan so that our "main" goroutine can continue.
// dispatch files
g.Go(func() error {
defer close(fileChan)
done := ctx.Done()
for _, file := range myFiles {
select {
case fileChan <- file:
continue
case <-done:
break
}
}
return ctx.Err()
})
I'm not sure it's strictly necessary to put this in the same errgroup in this case, because we can't get an error in the distributor goroutine. But this general pattern, drawn from the Pipeline example from errgroup, works regardless of whether the work dispatcher might generate errors.
This functions pretty simple, but the magic is in select along with ctx.Done() channel. Either we write to the work channel, or we fail if our context is done. This allows us to stop distributing work when one worker has failed one file.
We defer close(fileChan) so that, regardless of why we have finished (either we distributed all work, or the context was cancelled), the workers know there will be no more work on the incoming work queue (ie fileChan).
We need one more synchronization mechanism: once all the work is distributed, and all the results are in or work was finished being cancelled, (eg, after our errgroup's Wait() returns), we need to close our results channel, dataChan. This signals the results collector that there are no more results to be collected.
var err error // we'll need this later!
go func() {
err = g.Wait()
close(dataChan)
}()
We can't - and don't need to - put this in the errgroup.Group. The function can't return an error, and it can't wait for itself to close(dataChan). So it goes into a regular old goroutine, sans errgroup.
Finally we can collect the results. With dedicated worker goroutines, a distributor goroutine, and a goroutine waiting on the work and notifying that there will be no more writes to the dataChan, we can collect all the results right in the "primary" goroutine in main.
for data := range dataChan {
myMap[data.name] = data.bytes
}
if err != nil { // this was set in our final goroutine, remember
fmt.Println("errgroup Error:", err.Error())
}
I made a few small changes so that it was easier to see the output. You may already have noticed I changed the file contents from []byte to string. This was purely so that the results were easy to read. Pursuant also to that end, I am using encoding/json to format the results so that it is easy to read them and paste them into SO. This is a common pattern that I often use to indent structured data:
enc := json.NewEncoder(os.Stdout)
enc.SetIndent("", " ")
if err := enc.Encode(myMap); err != nil {
panic(err)
}
Finally we're ready to run. Now we can see a number of different results depending on just what order the goroutines execute. But all of them are valid execution paths.
worker 2 failed to process file2 : file2 cannot be read
worker 0 context error in worker: context canceled
worker 1 context error in worker: context canceled
errgroup Error: file2 cannot be read
{
"file1": "these are some bytes for file file1",
"file3": "these are some bytes for file file3"
}
Program exited.
In this result, the remaining work (file4 and file5) were not added to the channel. Remember, an unbuffered channel stores no data. For those tasks to be written to the channel, a worker would have to be there to read them. Instead, the context was cancelled after file2 failed, and the distribution function followed the <-done path within its select. file1 and file3 were already processed.
Here's a different result (I just ran the playground share a few times to get different results).
worker 1 failed to process file2 : file2 cannot be read
worker 2 processed all work on channel
worker 0 processed all work on channel
errgroup Error: file2 cannot be read
{
"file1": "these are some bytes for file file1",
"file3": "these are some bytes for file file3",
"file4": "these are some bytes for file file4",
"file5": "these are some bytes for file file5",
"file6": "these are some bytes for file file6"
}
In this case, it looks a little like our cancellation failed. but what really happened is that the goroutines just "happened" to queue and finish the rest of the work before errorgroup picked upon worker `'s failure and cancelled the context.
what errorgroup does
When you use errorgroup, you're really getting 2 things out of it:
easily accessing the first error your workers returned;
getting a context that errorgroup will cancel for you when
Keep in mind that errorgroup does not cancel goroutines. This tripped me up a bit at first. Errorgroup cancels the context. It's your responsibility to apply the status of that context to your goroutines (remember, the goroutine must end itself, errorgroup can't end it).
A final aside about contexts with file operations, and failing outstanding work
Most of your file operations, eg io.Copy or os.ReadFile, are actually a loop of subsequent Read operations. But io and os don't support contexts directly. so if you have a worker reading a file, and you don't implement the Read loop yourself, you won't have an opportunity to cancel based on context. That's probably okay in your case - sure, you may have read some more files than you really needed to, but only because you were already reading them when the error occurred. I would personally accept this state of affairs and not implement my own read loop.
The code
https://go.dev/play/p/9qfESp_eB-C
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"golang.org/x/sync/errgroup"
)
func main() {
var myFiles = []string{"file1", "file2", "file3", "file4", "file5", "file6"}
fileChan := make(chan string)
dataChan := make(chan fileData)
g, ctx := errgroup.WithContext(context.Background())
for i := 0; i < 3; i++ {
worker_num := i
g.Go(func() error {
for file := range fileChan {
if err := getBytesFromFile(file, dataChan); err != nil {
fmt.Println("worker", worker_num, "failed to process", file, ":", err.Error())
return err
} else if err := ctx.Err(); err != nil {
fmt.Println("worker", worker_num, "context error in worker:", err.Error())
return err
}
}
fmt.Println("worker", worker_num, "processed all work on channel")
return nil
})
}
// dispatch files
g.Go(func() error {
defer close(fileChan)
done := ctx.Done()
for _, file := range myFiles {
if err := ctx.Err(); err != nil {
return err
}
select {
case fileChan <- file:
continue
case <-done:
break
}
}
return ctx.Err()
})
var err error
go func() {
err = g.Wait()
close(dataChan)
}()
var myMap = make(map[string]string)
for data := range dataChan {
myMap[data.name] = data.bytes
}
if err != nil {
fmt.Println("errgroup Error:", err.Error())
}
enc := json.NewEncoder(os.Stdout)
enc.SetIndent("", " ")
if err := enc.Encode(myMap); err != nil {
panic(err)
}
}
type fileData struct {
name,
bytes string
}
func getBytesFromFile(file string, dataChan chan fileData) error {
bytes, err := openFileAndGetBytes(file)
if err == nil {
dataChan <- fileData{name: file, bytes: bytes}
}
return err
}
func openFileAndGetBytes(file string) (string, error) {
if file == "file2" {
return "", fmt.Errorf("%s cannot be read", file)
}
return fmt.Sprintf("these are some bytes for file %s", file), nil
}
I know there is a function called SetReadDeadline that can set a timeout in socket(conn.net) reading, while io.Read not. There is a way that starts another routine as a timer to solve this problem, but it brings another problem that the reader routine(io.Read) still block:
func (self *TimeoutReader) Read(buf []byte) (n int, err error) {
ch := make(chan bool)
n = 0
err = nil
go func() { // this goroutime still exist even when timeout
n, err = self.reader.Read(buf)
ch <- true
}()
select {
case <-ch:
return
case <-time.After(self.timeout):
return 0, errors.New("Timeout")
}
return
}
This question is similar in this post, but the answer is unclear.
Do you guys have any good idea to solve this problem?
Instead of setting a timeout directly on the read, you can close the os.File after a timeout. As written in https://golang.org/pkg/os/#File.Close
Close closes the File, rendering it unusable for I/O. On files that support SetDeadline, any pending I/O operations will be canceled and return immediately with an error.
This should cause your read to fail immediately.
Your mistake here is something different:
When you read from the reader you just read one time and that is wrong:
go func() {
n, err = self.reader.Read(buf) // this Read needs to be in a loop
ch <- true
}()
Here is a simple example (https://play.golang.org/p/2AnhrbrhLrv)
buf := bytes.NewBufferString("0123456789")
r := make([]byte, 3)
n, err := buf.Read(r)
fmt.Println(string(r), n, err)
// Output: 012 3 <nil>
The size of the given slice is used when using the io.Reader. If you would log the n variable in your code you would see, that not the whole file is read. The select statement outside of your goroutine is at the wrong place.
go func() {
a := make([]byte, 1024)
for {
select {
case <-quit:
result <- []byte{}
return
default:
_, err = self.reader.Read(buf)
if err == io.EOF {
result <- a
return
}
}
}
}()
But there is something more! You want to implement the io.Reader interface. After the Read() method is called until the file ends you should not start a goroutine in here, because you just read chunks of the file.
Also the timeout inside the Read() method doesn't help, because that timeout works for each call and not for the whole file.
In addition to #apxp's point about looping over Read, you could use a buffer size of 1 byte so that you never block as long is there is data to read.
When interacting with external resources anything can happen. It is possible for any given io.Reader implementation to simply block forever. Here, I'll write one for you...
type BlockingReader struct{}
func (BlockingReader) Read(b []byte) (int, error) {
<-make(chan struct{})
return 0, nil
}
Remember anyone can implement an interface, so you can't make any assumptions that it will behave like *os.File or any other standard library io.Reader. In addition to asinine coding like mine above, an io.Reader could legitimately connect to a resources that can block forever.
You cannot kill gorountines, so if an io.Reader truly blocks forever the blocked goroutine will continue to consume resources until your application terminates. However, this shouldn't be a problem, a blocked goroutine does not consume much in the way of resources, and should be fine as long as you don't blindly retry blocked Reads by spawning more gorountines.
It doesn't seem possible to have two way communication via channels with a goroutine which is performing file operations, unless you block the channel communication on the file operations. How can I work around the limits this imposes?
Another way to phrase this question...
If I have a loop similar to the following running in a goroutine, how can I tell it to close the connection and exit without blocking on the next Read?
func readLines(response *http.Response, outgoing chan string) error {
defer response.Body.Close()
reader := bufio.NewReader(response.Body)
for {
line, err := reader.ReadString('\n')
if err != nil {
return err
}
outgoing <- line
}
}
It's not possible for it to read from a channel that tells it when to close down because it's blocking on the network reads (in my case, that can take hours).
It doesn't appear to be safe to simply call Close() from outside the goroutine, since the Read/Close methods don't appear to be fully thread safe.
I could simply put a lock around references to response.Body that used inside/outside the routine, but would cause the external code to block until a pending read completes, and I specifically want to be able to interrupt an in-progress read.
To address this scenario, several io.ReadCloser implementations in the standard library support concurrent calls to Read and Close where Close interrupts an active Read.
The response body reader created by net/http Transport is one of those implementations. It is safe to concurrently call Read and Close on the response body.
You can also interrupt an active Read on the response body by calling the Transport CancelRequest method.
Here's how implement cancel using close on the body:
func readLines(response *http.Response, outgoing chan string, done chan struct{}) error {
cancel := make(chan struct{})
go func() {
select {
case <-done:
response.Body.Close()
case <-cancel:
return
}()
defer response.Body.Close()
defer close(cancel) // ensure that goroutine exits
reader := bufio.NewReader(response.Body)
for {
line, err := reader.ReadString('\n')
if err != nil {
return err
}
outgoing <- line
}
}
Calling close(done) from another goroutine will cancel reads on the body.
The problem is this: There is a web server. I figured that it would be beneficial to use goroutines in page loading, so I went ahead and did: called loadPage function as a goroutine. However, when doing this, the server simply stops working without errors. It prints a blank, white page. The problem has to be in the function itself- something there is conflicting with the goroutine somehow.
These are the relevant functions:
func loadPage(w http.ResponseWriter, path string) {
s := GetFileContent(path)
w.Header().Add("Content-Type", getHeader(path))
w.Header().Add("Content-Length", GetContentLength(path))
fmt.Fprint(w, s)
}
func GetFileContent(path string) string {
cont, err := ioutil.ReadFile(path)
e(err)
aob := len(cont)
s := string(cont[:aob])
return s
}
func GetFileContent(path string) string {
cont, err := ioutil.ReadFile(path)
e(err)
aob := len(cont)
s := string(cont[:aob])
return s
}
func getHeader(path string) string {
images := []string{".jpg", ".jpeg", ".gif", ".png"}
readable := []string{".htm", ".html", ".php", ".asp", ".js", ".css"}
if ArrayContainsSuffix(images, path) {
return "image/jpeg"
}
if ArrayContainsSuffix(readable, path) {
return "text/html"
}
return "file/downloadable"
}
func ArrayContainsSuffix(arr []string, c string) bool {
length := len(arr)
for i := 0; i < length; i++ {
s := arr[i]
if strings.HasSuffix(c, s) {
return true
}
}
return false
}
The reason why this happens is because your HandlerFunc which calls "loadPage" is called synchronously with the request. When you call it in a go routine the Handler is actually returning immediately, causing the response to be sent immediately. That's why you get a blank page.
You can see this in server.go (line 1096):
serverHandler{c.server}.ServeHTTP(w, w.req)
if c.hijacked() {
return
}
w.finishRequest()
The ServeHTTP function calls your handler, and as soon as it returns it calls "finishRequest". So your Handler function must block as long as it wants to fulfill the request.
Using a go routine will actually not make your page any faster. Synchronizing a singe go routine with a channel, as Philip suggests, will also not help you in this case as that would be the same as not having the go routine at all.
The root of your problem is actually ioutil.ReadFile, which buffers the entire file into memory before sending it.
If you want to stream the file you need to use os.Open. You can use io.Copy to stream the contents of the file to the browser, which will used chunked encoding.
That would look something like this:
f, err := os.Open(path)
if err != nil {
http.Error(w, "Not Found", http.StatusNotFound)
return
}
n, err := io.Copy(w, f)
if n == 0 && err != nil {
http.Error(w, "Error", http.StatusInternalServerError)
return
}
If for some reason you need to do work in multiple go routines, take a look at sync.WaitGroup. Channels can also work.
If you are trying to just serve a file, there are other options that are optimized for this, such as FileServer or ServeFile.
In the typical web framework implementations in Go, the route handlers are invoked as Goroutines. I.e. at some point the web framework will say go loadPage(...).
So if you call a Go routine from inside loadPage, you have two levels of Goroutines.
The Go scheduler is really lazy and will not execute the second level if it's not forced to. So you need to enforce it through synchronization events. E.g. by using channels or the sync package. Example:
func loadPage(w http.ResponseWriter, path string) {
s := make(chan string)
go GetFileContent(path, s)
fmt.Fprint(w, <-s)
}
The Go documentation says this:
If the effects of a goroutine must be observed by another goroutine,
use a synchronization mechanism such as a lock or channel
communication to establish a relative ordering.
Why is this actually a smart thing to do? In larger projects you may deal with a large number of Goroutines that need to be coordinated somehow efficiently. So why call a Goroutine if it's output is used nowhere? A fun fact: I/O operations like fmt.Printf do trigger synchronization events too.