This is the code to upload an apk file (several MB) to appetize.io using golang:
func uploadToAppetize(file multipart.File, branchName string, displayName string) (result *AppetizeRes, ok bool) {
file.Seek(0, 0)
url, _ := getUrl()
var buffer bytes.Buffer
writer := multipart.NewWriter(&buffer)
fileName := displayName + "/" + branchName
part, err := writer.CreateFormFile("file", fileName)
if err != nil {
fmt.Fprintf(os.Stderr, "Error creating form file %v: %v\n", fileName, err)
return nil, false
}
size, err := io.Copy(part, file)
if err != nil {
fmt.Fprintf(os.Stderr, "Error copying apk file data: %v\n", err)
return nil, false
}
fmt.Fprintf(os.Stdout, "Copied %v bytes for uploading to appetize...\n", size)
writer.Close()
response, err := http.Post(url, writer.FormDataContentType(), &buffer) // Random error on this line
if err != nil || response == nil {
fmt.Fprintf(os.Stderr, "Error occurred uploading apk data to appetize.io: %v %v\n", err, response)
return nil, false
}
defer response.Body.Close()
if response.StatusCode != http.StatusOK {
return nil, false
}
var appetizeRes AppetizeRes
if err := json.NewDecoder(response.Body).Decode(&appetizeRes); err != nil {
return nil, false
}
return &appetizeRes, true
}
However I am receiving an random error on the line http.Post(...). It returns a nil response and an error. The error is "stream error: stream ID 1; REFUSED_STREAM". It happens randomly but will surely happen the first time the go program make the request after launching.
This is the go version:
go version go1.8.1 darwin/amd64
This is the response header from the server if it doesn't fail:
I also run this program on another mac running go 1.6.*, I didn't remember I ever run into this issue on that mac.
Any idea what's going on?
There is a bug in the golang standard library net/http that fails to handle the REFUSED_STREAM http/2 error properly. This is what is likely going on:
golang client opens a TCP connection to the HTTP/2 www server, sets the maximum number of HTTP streams in the connection to 1000 and starts uploading immediately.
The HTTP/2 www server tells the golang client to only use a given number of streams, but the golang client has already started more than that amount of streams.
The HTTP/2 www server reacts to this by resetting the excess streams.
The lack of proper support for a retry in the golang net/http code causes the stream to fail after being reset by the server, and eventually the upload fails as well.
There is a ticket open about this issue on github: x/net/http2: retry requests rejected with REFUSED_STREAM - golang/go/issues/20985
**This issue has been closed in the master branch, the fix will be included in future Golang version 1.10
Related
I am currently querying Druid and returning a large dataset back (roughly 4-5GBs). I would like to process this response and decode the JSON into a list of structs. I have it working fine when I change the query to return a smaller dataset but as soon as the response gets too large I get an unexpected EOF error.
I have tried reading the entire response body
bytes, err := ioutil.ReadAll(resp.Body)
Decoding the response body directly
var object []NewObject
err = json.NewDecoder(resp.Body).Decode(&object)
Creating a buffer and writing to a file
f, err := os.OpenFile("/tmp/test.txt", os.O_APPEND|os.O_WRONLY, 0600)
defer f.Close()
const oneMB = 1024 * 1024
bytesRead := 0
respBuf := make([]byte, oneMB)
// Read the response body
for {
n, err := resp.Body.Read(respBuf)
bytesRead += n
if err == io.EOF {
break
}
if err != nil {
fmt.Println("Error reading HTTP response: ", err.Error())
break
}
if _, err = f.Write(respBuf); err != nil {
panic(err)
}
}
All of these have ended with me getting an unexpected EOF error. I am using the default net/http, and encoding/json module which appear they should work fine. Is there anything else I can try?
I've figured out the problem. It appears to be that Druid has a hard query timeout (this link explains more about these configuration settings). What is interesting is it appears if you want the response as an object and it is too large it will just cut off mid byte stream, but if you want the result as array and it is too large it will send the bytes to finish the current element and then send a closing ] so when you parse the result it appears it all came through (no errors), though you may have been cut off midstream.
I have a service that use to upload file to AWS S3. I was trying to use with goroutines and without to upload the file. If I upload the file without goroutines, it should wait till finish then give the response, and if I use goroutines it will run in the background and faster to response to the client-side.
How about if that upload failed if I use goroutines? And then that file not uploaded to AWS S3? Can you tell me to handle this how?
here is my function to upload file
func uploadToS3(s *session.Session, size int64, name string , buffer []byte)( string , error) {
tempFileName := "pictures/" + bson.NewObjectId().Hex() + "-" + filepath.Base(name)
_, err := s3.New(s).PutObject(&s3.PutObjectInput{
Bucket: aws.String("myBucketNameHere"),
Key: aws.String(tempFileName),
ACL: aws.String("public-read"),
Body: bytes.NewReader(buffer),
ContentLength: aws.Int64(int64(size)),
ContentType: aws.String(http.DetectContentType(buffer)),
ContentDisposition: aws.String("attachment"),
ServerSideEncryption: aws.String("AES256"),
StorageClass: aws.String("INTELLIGENT_TIERING"),
})
if err != nil {
return "", err
}
return tempFileName, err
}
func UploadFile(db *gorm.DB) func(c *gin.Context) {
return func(c *gin.Context) {
file, err := c.FormFile("file")
f, err := file.Open()
if err != nil {
fmt.Println(err)
}
defer f.Close()
buffer := make([]byte, file.Size)
_, _ = f.Read(buffer)
s, err := session.NewSession(&aws.Config{
Region: aws.String("location here"),
Credentials: credentials.NewStaticCredentials(
"id",
"key",
"",
),
})
if err != nil {
fmt.Println(err)
}
go uploadToS3(s, file.Size, file.Filename, buffer)
c.JSON(200, fmt.Sprintf("Image uploaded successfully"))
}
}
I was thinking as well how about if there many request to upload a file over 10000+ per 5-10mins ? would some file can't be upload because too many request?
The problem is that when using a goroutine, you immediately return a success message to your client. If that's really the case, it means your goroutine needs to be able to recover in case of error when uploading to S3 (don't lose the image). So either you take care of that, or you inform asynchronously your client that the upload failed, so the client can re-try.
This question is too broad for a single answer. There are, broadly speaking, three possible approaches:
Wait for your goroutines to complete to handle any errors.
Ensure your goroutines can handle (or possibly ignore) any errors they encounter, such that returning an error never matters.
Have your goroutines log any errors, for handling later, possibly by a human, or possibly by some cleanup/retry function.
Which approach is best depends on the situation.
For any asynchronous task - such as uploading a file in a background go-routine - one can write the uploading function in such a way to return a chan error to the caller. The caller can then react to the file uploads eventual error (or nil for no error) at a later time by reading from the chan error.
However if you are accepting upload requests, I'd suggest instead to created a worker upload go-routine, that accepts file uploads via a channel. An output "error" channel can track success/failure. And if need be, the error uploaded could be written back to the original upload channel queue (including a retry tally & retry max - so a problematic payload does not loop forever) .
I'm using Go Gob to transfer large files (~ 1 GB) or many small files (~ 30 MB). Server is running in a loop, and will receive files when clients send it.
My code is working if I send one large file, or few small files, but when sending a large file for the second time, it returns a 'fatal error: runtime: out of memory'. If I send a large file, stops the program, then starts again and send another large file, it works.
It looks after receiving file via Gob and writing into a file, it is not releasing memory.
Server code
type FileGob struct {
FileName string
FileLen int
FileContent []byte
}
func handleConnection(conn net.Conn) {
transf := &FileGob{}
dec := gob.NewDecoder(conn)
err := dec.Decode(transf) // file from conn to var transf
if err != nil {
fmt.Println("error to decode into buffer:", err)
}
conn.Close()
file, err := os.Create("sent_" + transf.FileName)
if err != nil {
fmt.Println("error to create file:", err)
}
file.Write(transf.FileContent) // writes into file
fileStat, err := file.Stat()
if err != nil {
fmt.Println("error to get File Stat:", err)
}
file.Close()
fmt.Printf("File %v was transferred\n", transf.FileName)
fmt.Printf("Transferred: %d, Expected: %d\n", fileStat.Size(), transf.FileLen)
}
I have a test function which both creates a server and spawns a goroutine acting as a client. Now, simply sending a message from the client to the server works, but if I want to create an exchange, they seem to deadlock since the test never runs to completion (if no r/w deadlines are set). For example, I want the client to send a message to the server, the server to copy that message and send it back to the client, and then the client to verify that the received message was identical. Here is my test code:
func TestSendAwait(t *testing.T) {
m := "Hello World"
go func() {
conn, err := net.Dial("tcp", testingAddr)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
t.Log("client connected to server") // DEBUG
conn.SetDeadline(time.Now().Add(2 * time.Second))
conn.Write([]byte(m))
conn.SetDeadline(time.Now().Add(2 * time.Second))
buf, err := ioutil.ReadAll(conn)
if err != nil {
t.Fatal(err)
}
t.Log(string(buf))
}()
ln, err := net.Listen("tcp", testingAddr)
if err != nil {
t.Fatal(err)
}
defer ln.Close()
t.Log("server started") // DEBUG
conn, err := ln.Accept()
if err != nil {
t.Fatal(err)
}
defer conn.Close()
t.Log("server received connection") // DEBUG
buf, err := ioutil.ReadAll(conn)
if err != nil {
t.Fatal(err)
}
t.Logf("server read buffer: %v", buf) // DEBUG
_, err = conn.Write(buf)
if err != nil {
t.Fatal(err)
}
t.Log("server wrote to connection") // DEBUG
}
The deadlines are set on the connection because otherwise the deadlock would be indefinite. The output is as follows:
transmission_test.go:42: server started
transmission_test.go:24: client connected to server
transmission_test.go:49: server received connection
transmission_test.go:32: read tcp 127.0.0.1:41164->127.0.0.1:9090: i/o timeout
transmission_test.go:55: server read buffer: [72 101 108 108 111 32 87 111 114 108 100]
transmission_test.go:61: server wrote to connection
Process finished with exit code 1
I don't understand why the client is unable to read and exits, and only then the server decides to send data down the socket? This happens even if I increase the read deadline in the client.
The program blocks on the call to ioutil.ReadAll. This function reads until io.EOF or some other error is returned.
One fix is to shutdown write after writing data to the connection. This will cause read on the peer to return io.EOF and for ioutil.ReadAll to return successfully.
conn.Write(data)
cw, ok := conn.(interface{ CloseWrite() error })
if !ok {
// handle error
}
cw.CloseWrite()
playground example
The program in the question does not guarantee that the listener is opened before the connection is dialed or that client will print print the received message. The playground example corrects these issues.
Another approach is to frame the messages in some way:
Write newline or some other byte sequence not allowed in message after message. Read until this byte sequence is found.
Write message length before message. Read length and then specified number of bytes.
Here's the error I get when I flood the server with too many packets per second:
2014/11/28 12:52:49 main.go:59: loading plugin: print
2014/11/28 12:52:49 main.go:86: starting server on 0.0.0.0:8080
2014/11/28 12:52:59 server.go:15: client has connected: 127.0.0.1:59146
2014/11/28 12:52:59 server.go:43: received data from client 127.0.0.1:59146: &main.Observation{SensorId:"1", Timestamp:1416492023}
2014/11/28 12:52:59 server.go:29: read error from 127.0.0.1:59146: zlib: invalid header
2014/11/28 12:52:59 server.go:18: closing connection to: 127.0.0.1:59146
It manages to decode one packet (sometimes, maybe 2 or 3) then errors out. Here's the code doing the flooding:
import socket
import struct
import json
import zlib
import time
def serialize(data):
data = json.dumps(data)
data = zlib.compress(data)
packet = struct.pack('!I', len(data))
packet += data
return len(data), packet
message = {
'sensor_id': '1',
'timestamp': 1416492023,
}
length, buffer = serialize([message])
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(('127.0.0.1', 8080))
while True:
client.send(buffer)
#time.sleep(0.0005)
When I uncomment the time.sleep() call, the server works fine. It seems too many packets/per second is killing the server. Why?
Here's the relevent Go code. The connection handler:
func (self *Server) handleConnection(connection net.Conn) {
for {
connection.SetReadDeadline(time.Now().Add(30 * time.Second))
observations, err := self.Protocol.Unserialize(connection)
if err != nil {
log.Printf("read error from %s: %s\n", connection.RemoteAddr(), err)
return
}
}
And here's the unserializer:
// Length Value protocol to read zlib compressed, JSON encoded packets.
type ProtocolV2 struct{}
func (self *ProtocolV2) Unserialize(packet io.Reader) ([]*Observation, error) {
var length uint32
if err := binary.Read(packet, binary.BigEndian, &length); err != nil {
return nil, err
}
buffer := make([]byte, length)
rawreader := bufio.NewReader(packet)
if _, err := rawreader.Read(buffer); err != nil {
return nil, err
}
bytereader := bytes.NewReader(buffer)
zreader, err := zlib.NewReader(bytereader)
if err != nil {
return nil, err
}
defer zreader.Close()
var observations []*Observation
decoder := json.NewDecoder(zreader)
if err := decoder.Decode(&observations); err != nil {
return nil, err
}
return observations, nil
}
It seems there is an error on client side in the Python script.
The return of client.send is not checked, so the script does not handle partial writes in a correct way. Basically, when the socket buffer is full, only part of the message will be written, resulting in the server unable to decode the message.
This code is broken, but adding the wait makes it work because it prevents the socket buffer to be full.
You can use client.sendall instead to ensure the write operations are complete.
More information in the Python documentation:
https://docs.python.org/2/library/socket.html
https://docs.python.org/2/howto/sockets.html#using-a-socket
Now in the Go server, there is also a similar problem. The documentation says:
Read reads data into p. It returns the number of bytes read into p. It calls Read at most once on the underlying Reader, hence n may be less than len(p). At EOF, the count will be zero and err will be io.EOF.
The rawreader.Read call may return less bytes than you expect. You may want to use the ReadFull() function of the io package to ensure the full message is read.