HTTP API stops responding while writing file - performance

I have written an API http server in Go using Gorilla Mux. It works well. One of the endpoints is for uploading files and saving them to an NFS share mounted to the server pod. The client is a Swift 5 app using Alamofire.
For smaller files we just use copy to write them from the request body. For larger files, we use a buffered stream reader and writer to perform the write, as we had issues with time outs and drops when just using copy.
However, when the write is happening, the server stops responding to all new requests. How can I change or optimize this code so that the server continues to respond as expected? See code here:
func uploadFile(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
repoBase := "./repo/gkp-directory/"
folderName := vars["uploadFolder"]
fileName := vars["uploadFile"]
fileSize := r.ContentLength
//
// Check if we have a package. They can be large and require special handling
//
if folderName == "pkgs" {
defer r.Body.Close()
//
// If the content is more than 10MB write to temp cache then move it
// in a seperate goroutine to the repo storage
//
if r.ContentLength > 10000000 {
buf := make([]byte, 10000000)
tempFile, err := os.Create(repoBase + folderName + "/" + fileName)
if err != nil {
log.Println("ERROR: Failed to create file.")
log.Println(err.Error())
return
}
defer tempFile.Close()
for {
n, err := r.Body.Read(buf)
if err != nil && err != io.EOF {
log.Println("ERROR: Error creating file on NFS")
log.Println(err.Error())
return
}
if n == 0 {
break
}
if _, err := tempFile.Write(buf[:n]); err != nil {
log.Println("ERROR: Error streaming to NFS")
log.Println(err.Error())
return
}
}
tempFile.Close()
r.Body.Close()
} else {
//
// If the package is smaller than 10MB we should be safe to write it directly to
// the NFS backend with no buffer
//
outputFile, err := os.Create(repoBase + folderName + "/" + fileName)
if err != nil {
log.Println("ERROR: Failed to create file.")
log.Println(err.Error())
return
}
defer outputFile.Close()
written, err := io.Copy(outputFile, r.Body)
if err != nil {
log.Println("ERROR: Failed to create file.")
log.Println(err.Error())
return
}
if written == fileSize {
outputFile.Close()
r.Body.Close()
}
}
} else {
//
// Otherwise file is not a package. This means it is just a small text file we
// should safetly be able to write this to NFS with no issues or buffer
//
outputFile, err := os.Create(repoBase + folderName + fileName)
if err != nil {
log.Println("ERROR: Failed to create file.")
log.Println(err.Error())
return
}
defer outputFile.Close()
written, err := io.Copy(outputFile, r.Body)
if err != nil {
log.Println("ERROR: Failed to create file.")
log.Println(err.Error())
return
}
if written == fileSize {
outputFile.Close()
r.Body.Close()
}
}
}

Related

Transfering file using tcp golang

I'm trying to make a music app that sends file through tcp protocol using go and microservice architecture. Now I'm creating a player service that should:
Get user token and get claims from it
Check is user exists using claims and user_service microservice
Get song from redis
Check is song exists using music_service
Read file by chunks and send it to client using tcp
Redis data looks like this:
{
"user_id": [{
"song_id": "<song_id>"
}]
}
But I faced with a small problem. My music files stored in a flac format and when I receive it on the client, my player doesn't play it. I don't really know what can be the problem. So here's my code:
SERVER
service_setup.go
//this function is called in main function
func setService() {
ln, err := net.Listen("tcp", config.TCPAddress)
if err != nil {
panic("couldn't start tcp server")
}
defer ln.Close()
for {
conn, err := ln.Accept()
if err != nil {
logger.ErrorLog(fmt.Sprintf("Error: couldn't accept connection. Details: %v", err))
return
}
service.DownloadSong(conn)
}
}
downloader_service.go
func DownloadSong(conn net.Conn) {
token, err := bufio.NewReader(conn).ReadString('\n')
if err != nil {
logger.ErrorLog(fmt.Sprintf("Error: couldn't get token. Details: %v", token))
conn.Close()
return
}
claims, err := jwt_funcs.DecodeJwt(token)
if err != nil {
conn.Close()
return
}
songs, err := redis_repo.Get(claims.Id)
if err != nil {
conn.Close()
return
}
for _, song := range songs {
download(song, conn)
}
}
func download(song models.SongsModel, conn net.Conn) {
filePath, err := filepath.Abs(fmt.Sprintf("./songs/%s.flac", song.SongId))
if err != nil {
logger.ErrorLog(fmt.Sprintf("Errror: couldn't create filepath. Details: %v", err))
conn.Close()
return
}
file, err := os.Open(filePath)
defer file.Close()
if err != nil {
logger.ErrorLog(fmt.Sprintf("Errror: couldn't open file. Details: %v", err))
conn.Close()
return
}
read(file, conn)
}
func read(file *os.File, conn net.Conn) {
reader := bufio.NewReader(file)
buf := make([]byte, 15)
defer conn.Close()
for {
_, err := reader.Read(buf)
if err != nil && err == io.EOF {
logger.InfoLog(fmt.Sprintf("Details: %v", err))
fmt.Println()
return
}
conn.Write(buf)
}
}
CLIENT
main.go
func main() {
conn, _ := net.Dial("tcp", "127.0.0.1:6060")
var glMessage []byte
text := "eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6IjYzYzlhNmE1OWI3ZmQyNTQ2ZjA4ZWEyYSIsInVzZXJuYW1lIjoiMTIiLCJleHAiOjE2NzQyMTE5ODl9.aarSDhrFF1df3i2pIRyjNxTfSHKObqLU3kHJiPreredIhLNCzs7z7jMgRHQIcLaIvCOECN7bX0OaSvKdW7VKsQ\n"
fmt.Fprint(conn, text)
reader := bufio.NewReader(conn)
b := make([]byte, 15)
c := 0
for i, _ := reader.Read(b); int(i) != 0; i, _ = reader.Read(b) {
c += i
glMessage = append(glMessage, b...)
}
os.WriteFile("./test.flac", glMessage, 0644)
}
If you know what can be the problem, please tell me. I'd really appreciate it!
It looks like you're trying to send the music file over the network in 15 byte chunks, which is likely not enough to play the song on the client side.
You can try increasing the chunk size, for example, to 8192 bytes. To do this, replace buf := make([]byte, 15) with buf := make([]byte, 8192).
Also, it's better to write the received data directly to the file rather than storing it in memory. You can do this by creating a file and using os.Create to write the received data to it:
file, err := os.Create("./test.flac")
if err != nil {
fmt.Println("Error: couldn't create file")
return
}
defer file.Close()
for {
i, err := reader.Read(buf)
if err != nil && err == io.EOF {
break
}
file.Write(buf[:i])
}
I believe that this can solve the issue.

Copy file from remote to byte[]

I'm trying to figure out how to implement copying files from remote and get the data []byte from the buffer.
I have succeeded in doing the implementation with the upload by referring to this guide: https://chuacw.ath.cx/development/b/chuacw/archive/2019/02/04/how-the-scp-protocol-works.aspx
Inside the go func there's the implementation of the upload process of the SCP but I have no idea how to change it.
Any advice ?
func download(con *ssh.Client, buf bytes.Buffer, path string,) ([]byte,error) {
//https://chuacw.ath.cx/development/b/chuacw/archive/2019/02/04/how-the-scp-protocol-works.aspx
session, err := con.NewSession()
if err != nil {
return nil,err
}
buf.WriteString("sudo scp -f " + path + "\n")
stdin, err := session.StdinPipe()
if err != nil {
return nil,err
}
go func() {
defer stdin.Close()
fmt.Fprint(stdin, "C0660 "+strconv.Itoa(len(content))+" file\n")
stdin.Write(content)
fmt.Fprint(stdin, "\x00")
}()
output, err := session.CombinedOutput("sudo scp -f " + path)
buf.Write(output)
if err != nil {
return nil,&DeployError{
Err: err,
Output: buf.String(),
}
}
session.Close()
session, err = con.NewSession()
if err != nil {
return nil,err
}
defer session.Close()
return output,nil
}
The sink side is significantly more difficult than the source side. Made an example which should get you close to what you want. Note that I have not tested this code, that the error handling is sub optimal and it only supports 1/4th the protocol messages SCP may use. So you will still need to do some work to get it perfect.
With all that said, this is what I came up with:
func download(con *ssh.Client, path string) ([]byte, error) {
//https://chuacw.ath.cx/development/b/chuacw/archive/2019/02/04/how-the-scp-protocol-works.aspx
session, err := con.NewSession()
if err != nil {
return nil, err
}
defer session.Close()
// Local -> remote
stdin, err := session.StdinPipe()
if err != nil {
return nil, err
}
defer stdin.Close()
// Request a file, note that directories will require different handling
_, err = stdin.Write([]byte("sudo scp -f " + path + "\n"))
if err != nil {
return nil, err
}
// Remote -> local
stdout, err := session.StdoutPipe()
if err != nil {
return nil, err
}
// Make a buffer for the protocol messages
const megabyte = 1 << 20
b := make([]byte, megabyte)
// Offset into the buffer
off := 0
var filesize int64
// SCP may send multiple protocol messages, so keep reading
for {
n, err := stdout.Read(b[off:])
if err != nil {
return nil, err
}
nl := bytes.Index(b[:off+n], []byte("\n"))
// If there is no newline in the buffer, we need to read more
if nl == -1 {
off = off + n
continue
}
// We read a full message, reset the offset
off = 0
// if we did get a new line. We have the full protocol message
msg := string(b[:nl])
// Send back 0, which means OK, the SCP source will not send the next message otherwise
_, err = stdin.Write([]byte("0\n"))
if err != nil {
return nil, err
}
// First char is the mode (C=file, D=dir, E=End of dir, T=Time metadata)
mode := msg[0]
if mode != 'C' {
// Ignore other messags for now.
continue
}
// File message = Cmmmm <length> <filename>
msgParts := strings.Split(msg, " ")
if len(msgParts) > 1 {
// Parse the second part <length> as an base 10 integer
filesize, err = strconv.ParseInt(msgParts[1], 10, 64)
if err != nil {
return nil, err
}
}
// The file message will be followed with binary data containing the file
break
}
// Wrap the stdout reader in a limit reader so we will not read more than the filesize
fileReader := io.LimitReader(stdout, filesize)
// Seed the bytes buffer with the existing byte slice, saves additional allocation if file <= 1mb
buf := bytes.NewBuffer(b)
// Copy the file into the bytes buffer
_, err = io.Copy(buf, fileReader)
return buf.Bytes(), err
}

How can I serve files while using GRPC

Is there any way how to serve files in Go with GRPC, like in gin-gonic's variant:
router.Static("/static", "/var/www")
You can't do it exactly like that.
But you can use the proto bytes type and put the file bytes in that field.
Also (as pointed out in the comments) with large files you should use streaming instead of a unary call. (most GRPC implementation have a limit of 4MB per message).
Proto example:
syntax = "proto3";
message Response {
bytes fileChunk = 1;
}
message Request {
string fileName = 1;
}
service TestService {
rpc Download(Request) returns (stream Response);
}
Server implementation example:
func (srv *Server) Download(req *pbgo.Request, responseStream pbgo.TestService_DownloadServer) error {
bufferSize := 64 *1024 //64KiB, tweak this as desired
file, err := os.Open(req.GetFileName())
if err != nil {
fmt.Println(err)
return err
}
defer file.Close()
buff := make([]byte, bufferSize)
for {
bytesRead, err := file.Read(buff)
if err != nil {
if err != io.EOF {
fmt.Println(err)
}
break
}
resp := &pbgo.Response{
FileChunk: buff[:bytesRead],
}
err = responseStream.Send(resp)
if err != nil {
log.Println("error while sending chunk:", err)
return err
}
}
return nil
}
Client would call it like this:
conn, err := grpc.Dial("localhost:9090", grpc.WithInsecure())
if err != nil {
log.Fatal("client could connect to grpc service:", err)
}
c := pbgo.NewTestServiceClient(conn)
fileStreamResponse, err := c.Download(context.TODO(), &pbgo.Request{
FileName: "test.txt",
})
if err != nil {
log.Println("error downloading:", err)
return
}
for {
chunkResponse, err := fileStreamResponse.Recv()
if err == io.EOF {
log.Println("received all chunks")
break
}
if err != nil {
log.Println("err receiving chunk:", err)
break
}
log.Printf("got new chunk with data: %s \n", chunkResponse.FileChunk)
}
If you need to be able to serve arbitrary files, you would need to handle which files you allow serving (say someone requests the file /etc/passwd or something).
Not sure what exactly is the use case here.

Cannot able to send Chunks using stream

I'm trying to use GRPC Client side stream by Image processing, I'm also newbie in GRPC stream, Here I will be creating the image in small chunks and send into the Server, Chunks are created but cannot able to send it. Finally I'm getting EOF error.
Here I attached my sample code any one can guide me thanks.
Example:
func (c *ClientGRPC) UploadFile(ctx context.Context) (stats stats.Stats, err error) {
var (
writing = true
buf []byte
n int
status *pb.UploadStatus
)
cwd, _ := os.Getwd()
templatePath := filepath.Join(cwd, "/unnamed.png")
file, err := os.Open(templatePath)
if err != nil {
err = errors.Wrapf(err,
"failed to open file %s",
file)
return
}
defer file.Close()
stream, err := c.client.Upload(ctx)
if err != nil {
err = errors.Wrapf(err,
"failed to create upload stream for file %s",
file)
return
}
defer stream.CloseSend()
buf = make([]byte, c.chunkSize)
for writing {
n, err = file.Read(buf)
if err != nil {
if err == io.EOF {
writing = false
err = nil
continue
}
err = errors.Wrapf(err,
"errored while copying from file to buf")
return
}
err = stream.Send(&pb.Chunk{
Content: buf[:n],
})
if err != nil {
err = errors.Wrapf(err,
"failed to send chunk via stream") //`Here, I'm getting EOF error`.
return
}
}
status, err = stream.CloseAndRecv()
if err != nil {
err = errors.Wrapf(err,
"failed to receive upstream status response")
return
}
if status.Code != pb.UploadStatusCode_Ok {
err = errors.Errorf(
"upload failed - msg: %s",
status.Message)
return
}
return
}
Output:
client=====> failed to send chunk via stream: EOF
If you are using grpc underneath, then run your program with the environment variable GRPC_GO_LOG_VERBOSITY_LEVEL=99 GRPC_GO_LOG_SEVERITY_LEVEL=info to get logs from grpc to debug deeper (i.e. it is a connection level problem, or stream level problem).

Reading more than 4096 bytes per chunk with part.Read

I'm trying to process a multipart file upload in small chunks to avoid storing the entire file in memory. The following function seems to solve this, however when passing a []byte as the destination for the part.Read() method, it reads the part in chunks of 4096 bytes instead of in chunks of the destination size (len([]byte)).
When opening a local file and Read()'ing it into a []byte of the same size, it uses the entire space available as expected. Thus I think it's something specific to the part.Reader(). However, I'm unable to find anything about a default or max size for that function.
For reference, the function is as follows:
func ReceiveFile(w http.ResponseWriter, r *http.Request) {
reader, err := r.MultipartReader()
if err != nil {
panic(err)
}
if reader == nil {
panic("Wrong media type")
}
buf := make([]byte, 16384)
fmt.Println(len(buf))
for {
part, err := reader.NextPart()
if err == io.EOF {
break
}
if err != nil {
panic(err)
}
var n int
for {
n, err = part.Read(buf)
if err == io.EOF {
break
}
if err != nil {
panic(err)
}
fmt.Printf("Read %d bytes into buf\n", n)
fmt.Println(len(buf))
}
n, err = part.Read(buf)
fmt.Printf("Finally read %d bytes into buf\n", n)
fmt.Println(len(buf))
}
The part reader does not attempt to fill the caller's buffer as allowed by the io.Reader contract.
The best way to handle this depends on the requirements of the application.
If you want to slurp the part into memory, then use ioutil.ReadAll:
for {
part, err := reader.NextPart()
if err == io.EOF {
break
}
if err != nil {
// handle error
}
p, err := ioutil.ReadAll(part)
if err != nil {
// handle error
}
// p is []byte with the contents of the part
}
If you want to copy the part to the io.Writer w, then use io.Copy:
for {
part, err := reader.NextPart()
if err == io.EOF {
break
}
if err != nil {
// handle error
}
w := // open a writer
_, err := io.Copy(w, part)
if err != nil {
// handle error
}
}
If you want to process fixed size chunks, then use io.ReadFull:
buf := make([]byte, chunkSize)
for {
part, err := reader.NextPart()
if err == io.EOF {
break
}
if err != nil {
// handle error
}
_, err := io.ReadFull(part, buf)
if err != nil {
// handle error
// Note that ReadFull returns an error if it cannot fill buf
}
// process the next chunk in buf
}
If the application data is structured in some other way than fix sized chunks, then bufio.Scanner might be of help.
Instead change the chunk size, why not use io.ReadFull ?
https://golang.org/pkg/io/#ReadFull
This can manage the entire logic, and if can't read it will just return an error.

Resources