Golang broken pipe - go

I have a code segment that is reading from a TCP connection and after the first few connections, the server is outputing broken pipe however no error is occurring in my go code. The server sending the messages is at its core the coda hale metrics library, more specifically the PickledGraphite class.
here is the Go code that is reading:
func handleConn(conn net.Conn, id int) {
fmt.Println("handleConn")
defer conn.Close()
buf := make([]byte, 0, 10240)
tmp := make([]byte, 256)
fmt.Printf("%v Reading...\n", id)
for {
n, err := conn.Read(tmp)
fmt.Printf("%v Read %v\n", id, n)
if err != nil {
fmt.Printf("%v Got err: %v\n", id, err)
if err != io.EOF {
fmt.Printf("%v read error: %v\n", id, err)
}
buf = append(buf, tmp[:n]...)
break
}
buf = append(buf, tmp[:n]...)
}
fmt.Printf("%v Done Reading\n", id)
// Do stuff with buf
}
func main() {
ln, err := net.Listen("tcp", ":5555")
if err != nil {
fmt.Println(err)
os.Exit(-1)
}
id := 1
for {
fmt.Println("getting connection\n")
conn, err := ln.Accept()
if err != nil {
fmt.Println(err)
break
}
conn.SetReadDeadline(time.Now().Add(20 * time.Second))
fmt.Println("Got connection")
go handleConn(conn, id)
id = id + 1
fmt.Println("sent handleConn\n")
}
}
My code is still running, I can still execute nc commands and see my code receive it, so I am not sure how I am losing the connection.
If I remove the conn.SetReadDeadline() line then what happens is my code no longer receives a EOF after the first message.
Thanks in advance

Related

Transfering file using tcp golang

I'm trying to make a music app that sends file through tcp protocol using go and microservice architecture. Now I'm creating a player service that should:
Get user token and get claims from it
Check is user exists using claims and user_service microservice
Get song from redis
Check is song exists using music_service
Read file by chunks and send it to client using tcp
Redis data looks like this:
{
"user_id": [{
"song_id": "<song_id>"
}]
}
But I faced with a small problem. My music files stored in a flac format and when I receive it on the client, my player doesn't play it. I don't really know what can be the problem. So here's my code:
SERVER
service_setup.go
//this function is called in main function
func setService() {
ln, err := net.Listen("tcp", config.TCPAddress)
if err != nil {
panic("couldn't start tcp server")
}
defer ln.Close()
for {
conn, err := ln.Accept()
if err != nil {
logger.ErrorLog(fmt.Sprintf("Error: couldn't accept connection. Details: %v", err))
return
}
service.DownloadSong(conn)
}
}
downloader_service.go
func DownloadSong(conn net.Conn) {
token, err := bufio.NewReader(conn).ReadString('\n')
if err != nil {
logger.ErrorLog(fmt.Sprintf("Error: couldn't get token. Details: %v", token))
conn.Close()
return
}
claims, err := jwt_funcs.DecodeJwt(token)
if err != nil {
conn.Close()
return
}
songs, err := redis_repo.Get(claims.Id)
if err != nil {
conn.Close()
return
}
for _, song := range songs {
download(song, conn)
}
}
func download(song models.SongsModel, conn net.Conn) {
filePath, err := filepath.Abs(fmt.Sprintf("./songs/%s.flac", song.SongId))
if err != nil {
logger.ErrorLog(fmt.Sprintf("Errror: couldn't create filepath. Details: %v", err))
conn.Close()
return
}
file, err := os.Open(filePath)
defer file.Close()
if err != nil {
logger.ErrorLog(fmt.Sprintf("Errror: couldn't open file. Details: %v", err))
conn.Close()
return
}
read(file, conn)
}
func read(file *os.File, conn net.Conn) {
reader := bufio.NewReader(file)
buf := make([]byte, 15)
defer conn.Close()
for {
_, err := reader.Read(buf)
if err != nil && err == io.EOF {
logger.InfoLog(fmt.Sprintf("Details: %v", err))
fmt.Println()
return
}
conn.Write(buf)
}
}
CLIENT
main.go
func main() {
conn, _ := net.Dial("tcp", "127.0.0.1:6060")
var glMessage []byte
text := "eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6IjYzYzlhNmE1OWI3ZmQyNTQ2ZjA4ZWEyYSIsInVzZXJuYW1lIjoiMTIiLCJleHAiOjE2NzQyMTE5ODl9.aarSDhrFF1df3i2pIRyjNxTfSHKObqLU3kHJiPreredIhLNCzs7z7jMgRHQIcLaIvCOECN7bX0OaSvKdW7VKsQ\n"
fmt.Fprint(conn, text)
reader := bufio.NewReader(conn)
b := make([]byte, 15)
c := 0
for i, _ := reader.Read(b); int(i) != 0; i, _ = reader.Read(b) {
c += i
glMessage = append(glMessage, b...)
}
os.WriteFile("./test.flac", glMessage, 0644)
}
If you know what can be the problem, please tell me. I'd really appreciate it!
It looks like you're trying to send the music file over the network in 15 byte chunks, which is likely not enough to play the song on the client side.
You can try increasing the chunk size, for example, to 8192 bytes. To do this, replace buf := make([]byte, 15) with buf := make([]byte, 8192).
Also, it's better to write the received data directly to the file rather than storing it in memory. You can do this by creating a file and using os.Create to write the received data to it:
file, err := os.Create("./test.flac")
if err != nil {
fmt.Println("Error: couldn't create file")
return
}
defer file.Close()
for {
i, err := reader.Read(buf)
if err != nil && err == io.EOF {
break
}
file.Write(buf[:i])
}
I believe that this can solve the issue.

Compress and transfer file via TCP (Golang)

I write simple example code, it worked, but size of file that recived is not compressed
My client (for connect to server and send file):
// connect to server
conn, err := net.Dial("tcp", serverAddr)
CheckError(err)
defer conn.Close()
in, err := os.Open(srcFile)
if err != nil {
log.Fatal(err)
}
pr, pw := io.Pipe()
gw, err := gzip.NewWriterLevel(pw, 7)
CheckError(err)
go func() {
n, err := io.Copy(gw, in)
gw.Close()
pw.Close()
log.Printf("copied %v %v", n, err)
}()
//maybe error some next?
_, err = io.Copy(conn, pr)
Please, help, how right to use pipe with copy
As I already said in the comment, your code works. I created a little example to test or see if I can solve your problem. So I guess you can close this question.
package main
import (
"compress/gzip"
"io"
"log"
"net"
"os"
)
func main() {
// Create a listener on a random port.
listener, err := net.Listen("tcp", "127.0.0.1:")
if err != nil {
log.Fatal(err)
}
log.Println("Server listening on: " + listener.Addr().String())
done := make(chan struct{})
go func() {
defer func() { done <- struct{}{} }()
for {
conn, err := listener.Accept()
if err != nil {
log.Println(err)
return
}
go func(c net.Conn) {
defer func() {
c.Close()
done <- struct{}{}
}()
buf := make([]byte, 1024)
for {
n, err := c.Read(buf)
if err != nil {
if err != io.EOF {
log.Println(err)
}
return
}
log.Printf("received: %q", buf[:n])
log.Printf("bytes: %d", n)
}
}(conn)
}
}()
conn, err := net.Dial("tcp", listener.Addr().String())
if err != nil {
log.Fatal(err)
}
log.Println("Connected to server.")
file, err := os.Open("./file.txt")
if err != nil {
log.Fatal(err)
}
pr, pw := io.Pipe()
w, err := gzip.NewWriterLevel(pw, 7)
if err != nil {
log.Fatal(err)
}
go func() {
n, err := io.Copy(w, file)
if err != nil {
log.Fatal(err)
}
w.Close()
pw.Close()
log.Printf("copied to piped writer via the compressed writer: %d", n)
}()
n, err := io.Copy(conn, pr)
if err != nil {
log.Fatal(err)
}
log.Printf("copied to connection: %d", n)
conn.Close()
<-done
listener.Close()
<-done
}
The output of that program with a simple text file with many repeated characters in it, to have something to compress: The file is 153 bytes and I send/received 46 bytes
2022/04/04 11:23:58 Server listening on: 127.0.0.1:58250
2022/04/04 11:23:58 Connected to server.
2022/04/04 11:23:58 received: "\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff"
2022/04/04 11:23:58 bytes: 10
2022/04/04 11:23:58 copied to piped writer via the compressed writer: 153
2022/04/04 11:23:58 copied to connection: 46
2022/04/04 11:23:58 received: "*I-.I,NI,N\xc1\x01\x8aS\x8a\x13i\bx\xb9pX \r\b\x00\x00\xff\xff\xc7\xfe\xa6c\x99\x00\x00\x00"
2022/04/04 11:23:58 bytes: 36
2022/04/04 11:23:58 accept tcp 127.0.0.1:58250: use of closed network connection

Golang io.copy not copying entire data

I have small code which reads 100 MB file from Google cloud storage and then return output.
Code works fine for 1MB file but fails for 100 mb file.
Below is the code which is not working
rc, err := client.Bucket("mabucket").Object(gcpurl).NewReader(ctx)
if err != nil {
fmt.Println(err)
return
}
defer rc.Close()
w.Header().Set("Content-Length", strconv.Itoa(int(rc.Size())))
w.Header().Set("Cache-Control", "max-age=2592000")
w.Header().Set("Content-Type", rc.ContentType())
spew.Dump(rc.ContentType())
if rc.ContentType() == "audio/wav" || rc.ContentType() == "audio/wave" {
w.Header().Set("Accept-Ranges", "bytes")
tilrange := rc.Size() - 1
newRangeString := "bytes 0-" + strconv.Itoa(int(tilrange)) + "/" + strconv.Itoa(int(rc.Size()))
w.Header().Set("Content-Range", newRangeString)
w.WriteHeader(206)
}
//spew.Dump(rc.Attrs)
io.Copy(w, rc)
I have written another code which is able to download same file and create a local file of 100 mb.
this time I am using ioutil.ReadAll. what can be problem with io.copy when receiving large date from GCP?
func main() {
ctx := context.Background()
client, _ := storage.NewClient(ctx)
data, err := downloadFile(client, "mabucket", "606ff2b71a916907409a953f/606ff2ed1a916907409a9540/60a38a967b291f7b44488824/123/audio/210415164000M29713363.wav")
//210415164000M29713363.wav
if err != nil {
log.Fatalf("Cannot read object: %v", err)
}
fmt.Printf("Object contents: %d\n", len(data))
f, err := os.Create("a.wav")
check(err)
defer f.Close()
n2, err := f.Write(data)
check(err)
fmt.Printf("wrote %d bytes\n", n2)
}
// downloadFile downloads an object.
func downloadFile(client *storage.Client, bucket, object string) ([]byte, error) {
// [START download_file]
ctx := context.Background()
ctx, cancel := context.WithTimeout(ctx, time.Second*50)
defer cancel()
rc, err := client.Bucket(bucket).Object(object).NewReader(ctx)
if err != nil {
return nil, err
}
defer rc.Close()
data, err := ioutil.ReadAll(rc)
if err != nil {
return nil, err
}
return data, nil
// [END download_file]
}
io.copy was trying to copy the details but OS was not allowing it. it was throwing error as (*net.OpError)(0xc0003302d0)(write tcp [::1]:80->[::1]:63014: wsasend: An established connection was aborted by the software in your host machine

Channels in Golang with TCP/IP socket not working

I just started writting a Golang client for a server that I've made in C with TCP/IP sockets, then I figured out that my channel wasn't working.
Any ideas why ?
func reader(r io.Reader, channel chan<- []byte) {
buf := make([]byte, 2048)
for {
n, err := r.Read(buf[:])
if err != nil {
return
}
channel <- buf[0:n]
}
}
func client(e *gowd.Element) {
f, err := os.Create("/tmp/dat2")
if err != nil {
log.Fatal()
}
read := make(chan []byte)
c, err := net.Dial("tcp", "127.0.0.1:4242")
if err != nil {
log.Fatal(err)
}
go reader(c, read)
for {
buf := <-read
n := strings.Index(string(buf), "\n")
if n == -1 {
continue
}
msg := string(buf[0:n])
if msg == "WELCOME" {
fmt.Fprint(c, "GRAPHIC\n")
}
f.WriteString(msg + "\n")
}
Testing my server with netcat results in the following output :
http://pasted.co/a37b2954
But i only have : http://pasted.co/f13d56b4
I'm new to chan in Golang so maybe I'm wrong (I probably am)
Channel usage looks alright, however retrieving value from channel would overwrite previously read value at buf := <-read since your waiting for newline.
Also you can use bufio.Reader to read string upto newline.
Your code snippet is partial so its not feasible to execute, try and let me know:
func reader(r io.Reader, channel chan<- string) {
bufReader := bufio.NewReader(conn)
for {
msg, err := bufReader.ReadString('\n')
if err != nil { // connection error or connection reset error, etc
break
}
channel <- msg
}
}
func client(e *gowd.Element) {
f, err := os.Create("/tmp/dat2")
if err != nil {
log.Fatal()
}
read := make(chan string)
c, err := net.Dial("tcp", "127.0.0.1:4242")
if err != nil {
log.Fatal(err)
}
go reader(c, read)
for {
msg := <-read
if msg == "WELCOME" {
fmt.Fprint(c, "GRAPHIC\n")
}
f.WriteString(msg + "\n")
}
//...
}
EDIT:
Please find example of generic TCP client to read data. Also I have removed scanner from above code snippet and added buffer reader.
func main() {
conn, err := net.Dial("tcp", "127.0.0.1:4242")
if err != nil {
log.Fatal(err)
}
reader := bufio.NewReader(conn)
for {
msg, err := reader.ReadString('\n')
if err != nil {
break
}
fmt.Println(msg)
}
}

Golang TCP Client does not receive data from server, hangs/blocks on conn.Read()

I'm taking a dive into the networking side of Go, and I'd thought I'd start with a TCP Client and Server.
I am able to get the client to connect to the server and send a simple message ("Hello") successfully. However, I can not get the server to send back a response (or the get the client to read the response).
Here is the code.
Server
Address := "localhost:9999"
Addr, err := net.ResolveTCPAddr("tcp", Address)
if err != nil {
log.Fatal(err)
}
listener, err := net.ListenTCP("tcp", Addr)
if err != nil {
log.Fatal(err)
}
defer listener.Close()
//server loop
for {
conn, err := listener.Accept()
if err != nil {
continue
}
go handle(conn)
}
func handle(c net.Conn) {
totalBytes, message := connRead(c)
fmt.Println(c.RemoteAddr())
fmt.Println(string(message[:totalBytes]))
c.Write([]byte("Hi"))
fmt.Println("Replied")
c.Close()
}
func connRead(c net.Conn) (int, []byte) {
buffer := make([]byte, 4096)
totalBytes := 0
for {
n, err := c.Read(buffer)
totalBytes += n
if err != nil {
if err != io.EOF {
log.Printf("Read error: %s", err)
}
break
}
}
return totalBytes, buffer
}
Client
tcpAddr, err := net.ResolveTCPAddr("tcp", "localhost:9999")
if err != nil {
log.Fatal(err)
}
conn, err := net.DialTCP("tcp", nil, tcpAddr)
if err != nil {
log.Fatal(err)
}
defer conn.Close()
_, err = conn.Write([]byte("Hello"))
if err != nil {
log.Fatal(err)
}
tBytes, resp := connRead(conn)
fmt.Println(tBytes)
fmt.Println(string(resp[:tBytes]))
func connRead(c net.Conn) (int, []byte) {
buffer := make([]byte, 4096)
totalBytes := 0
for {
fmt.Println("Stuck?")
n, err := c.Read(buffer)
fmt.Println("Stuck.")
totalBytes += n
fmt.Println(totalBytes)
if err != nil {
if err != io.EOF {
log.Printf("Read error: %s", err)
}
break
}
}
return totalBytes, buffer
}
From what I can tell it's not a problem with the server. When I run the client, everything stops right after fmt.Println("Stuck?"). This leads me to belive that it's messing up in the n, err := c.Read(buffer) statement somehow. The server doesn't even print out the messeage length (5) and message ("Hello") untill after I Ctrl-C the client. If I comment out the read and printings in the client, then things run smoothly.
I've tried googling for answers, but nothing has come up.
What am I doing wrong? Am I using conn.Read() wrong in the client?
EDIT:
I actually do have access to Linux, so here are the SIGQUIT dumps for the pertinent functions.
Server
http://pastebin.com/itevngCq
Client
http://pastebin.com/XLiKqkvs
for {
n, err := c.Read(buffer)
totalBytes += n
if err != nil {
if err != io.EOF {
log.Printf("Read error: %s", err)
}
break
}
}
It is because you are reading from connection till EOF error occurs
conn.Write([]byte("Hello"))
The above statement won't reach EOF at all until you actually closes the connection
On pressing ctrl+c client side the connection will be closed, So EOF occurs at server side, That is the reason why it is exiting server side for loop and printing these
127.0.0.1:****
Hello
Replied
If you want to make this work you should not read the connection till EOF
There are many other alternatives for this
Chose a delimiter and read at the server until the delimiter occurs and respond back after that. Check out this link
Send number of bytes to read from client side before sending the actual message, First read number of bytes to read from the server side and then read those many bytes from the connection

Resources