go routines hang on download of large files - go

Question

I am attempting to write an application that will download a range of images.
130 116kb images (works)
50 500kb images (works)
130 500kb images (eventually hangs)
230 116kb images (eventually hangs)
go version go1.9.2 darwin/amd64
package main
import (
"fmt"
"io"
"log"
"net/http"
"os"
"sync"
)
func main() {
var urls []string
// var smallImage = "https://s3.amazonaws.com/golangplayground/116kb.jpeg" //116kb
var largeImage = "https://s3.amazonaws.com/golangplayground/SampleJPGImage_500kbmb.jpg" //500kb
for i := 0; i < 130; i++ {
urls = append(urls, largeImage)
}
var wg sync.WaitGroup
wg.Add(len(urls))
var inc = 0
for _, val := range urls {
inc += 1
go saveResourceFromURLToDisk(val, "./foo", &wg, inc)
}
wg.Wait()
fmt.Println("done.")
}
func saveResourceFromURLToDisk(url string, writeTo string, wg *sync.WaitGroup, inc int) error {
defer wg.Done()
response, err := http.Get(url)
if err != nil {
log.Fatal(err)
return err
}
defer response.Body.Close()
localPath := fmt.Sprintf("%s/%d", writeTo, inc)
file, err := os.Create(localPath)
if err != nil {
log.Fatal(err)
return err
}
defer file.Close()
_, err = io.Copy(file, response.Body)
if err != nil {
log.Fatal(err)
return err
}
fmt.Println(localPath)
return nil
}

Answer 1

This is probably a network problem. There's a reason web browsers have limits on how many sessions to open to the same server.
If you open a bunch of TCP sessions all at the same time, almost all of them will lose packets. Then they'll all try to retry at about the same time, losing more packets. It is just a big pile of lose.
Place a small delay between opening each GET request or limit yourself to 4 - 8 simultaneous downloads from the same server.

Answer 2

I found the answer with Zan's help by bucketing my go routines in 5 requests each... This way I get to take advantage of some parallelism while throttling the amount of open connections I am creating.
Its a bit naive and I am wondering if anyone has a more elegant solution.
package main
import (
"fmt"
"io"
"log"
"net/http"
"os"
"sync"
)
func main() {
var urls []string
// var smallImage = "https://s3.amazonaws.com/golangplayground/116kb.jpeg" //116kb
var largeImage = "https://s3.amazonaws.com/golangplayground/SampleJPGImage_500kbmb.jpg" //500kb
for i := 0; i < 150; i++ {
urls = append(urls, largeImage)
}
var inc = 0;
for x:=0; x < len(urls)/5; x++ {
var wg sync.WaitGroup
for y:=0; y<5; y++ {
wg.Add(1)
go saveResourceFromURLToDisk(urls[x*y], "./foo", &wg, inc)
inc += 1
}
wg.Wait()
}
fmt.Println("done.")
}
func saveResourceFromURLToDisk(url string, writeTo string, wg *sync.WaitGroup, inc int) error {
defer wg.Done()
response, err := http.Get(url)
if err != nil {
log.Fatal(err)
return err
}
defer response.Body.Close()
localPath := fmt.Sprintf("%s/%d", writeTo, inc)
file, err := os.Create(localPath)
if err != nil {
log.Fatal(err)
return err
}
defer file.Close()
_, err = io.Copy(file, response.Body)
if err != nil {
log.Fatal(err)
return err
}
fmt.Println(localPath)
return nil
}