Compress and transfer file via TCP (Golang) - go

I write simple example code, it worked, but size of file that recived is not compressed
My client (for connect to server and send file):
// connect to server
conn, err := net.Dial("tcp", serverAddr)
CheckError(err)
defer conn.Close()
in, err := os.Open(srcFile)
if err != nil {
log.Fatal(err)
}
pr, pw := io.Pipe()
gw, err := gzip.NewWriterLevel(pw, 7)
CheckError(err)
go func() {
n, err := io.Copy(gw, in)
gw.Close()
pw.Close()
log.Printf("copied %v %v", n, err)
}()
//maybe error some next?
_, err = io.Copy(conn, pr)
Please, help, how right to use pipe with copy

As I already said in the comment, your code works. I created a little example to test or see if I can solve your problem. So I guess you can close this question.
package main
import (
"compress/gzip"
"io"
"log"
"net"
"os"
)
func main() {
// Create a listener on a random port.
listener, err := net.Listen("tcp", "127.0.0.1:")
if err != nil {
log.Fatal(err)
}
log.Println("Server listening on: " + listener.Addr().String())
done := make(chan struct{})
go func() {
defer func() { done <- struct{}{} }()
for {
conn, err := listener.Accept()
if err != nil {
log.Println(err)
return
}
go func(c net.Conn) {
defer func() {
c.Close()
done <- struct{}{}
}()
buf := make([]byte, 1024)
for {
n, err := c.Read(buf)
if err != nil {
if err != io.EOF {
log.Println(err)
}
return
}
log.Printf("received: %q", buf[:n])
log.Printf("bytes: %d", n)
}
}(conn)
}
}()
conn, err := net.Dial("tcp", listener.Addr().String())
if err != nil {
log.Fatal(err)
}
log.Println("Connected to server.")
file, err := os.Open("./file.txt")
if err != nil {
log.Fatal(err)
}
pr, pw := io.Pipe()
w, err := gzip.NewWriterLevel(pw, 7)
if err != nil {
log.Fatal(err)
}
go func() {
n, err := io.Copy(w, file)
if err != nil {
log.Fatal(err)
}
w.Close()
pw.Close()
log.Printf("copied to piped writer via the compressed writer: %d", n)
}()
n, err := io.Copy(conn, pr)
if err != nil {
log.Fatal(err)
}
log.Printf("copied to connection: %d", n)
conn.Close()
<-done
listener.Close()
<-done
}
The output of that program with a simple text file with many repeated characters in it, to have something to compress: The file is 153 bytes and I send/received 46 bytes
2022/04/04 11:23:58 Server listening on: 127.0.0.1:58250
2022/04/04 11:23:58 Connected to server.
2022/04/04 11:23:58 received: "\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff"
2022/04/04 11:23:58 bytes: 10
2022/04/04 11:23:58 copied to piped writer via the compressed writer: 153
2022/04/04 11:23:58 copied to connection: 46
2022/04/04 11:23:58 received: "*I-.I,NI,N\xc1\x01\x8aS\x8a\x13i\bx\xb9pX \r\b\x00\x00\xff\xff\xc7\xfe\xa6c\x99\x00\x00\x00"
2022/04/04 11:23:58 bytes: 36
2022/04/04 11:23:58 accept tcp 127.0.0.1:58250: use of closed network connection

Related

Transfering file using tcp golang

I'm trying to make a music app that sends file through tcp protocol using go and microservice architecture. Now I'm creating a player service that should:
Get user token and get claims from it
Check is user exists using claims and user_service microservice
Get song from redis
Check is song exists using music_service
Read file by chunks and send it to client using tcp
Redis data looks like this:
{
"user_id": [{
"song_id": "<song_id>"
}]
}
But I faced with a small problem. My music files stored in a flac format and when I receive it on the client, my player doesn't play it. I don't really know what can be the problem. So here's my code:
SERVER
service_setup.go
//this function is called in main function
func setService() {
ln, err := net.Listen("tcp", config.TCPAddress)
if err != nil {
panic("couldn't start tcp server")
}
defer ln.Close()
for {
conn, err := ln.Accept()
if err != nil {
logger.ErrorLog(fmt.Sprintf("Error: couldn't accept connection. Details: %v", err))
return
}
service.DownloadSong(conn)
}
}
downloader_service.go
func DownloadSong(conn net.Conn) {
token, err := bufio.NewReader(conn).ReadString('\n')
if err != nil {
logger.ErrorLog(fmt.Sprintf("Error: couldn't get token. Details: %v", token))
conn.Close()
return
}
claims, err := jwt_funcs.DecodeJwt(token)
if err != nil {
conn.Close()
return
}
songs, err := redis_repo.Get(claims.Id)
if err != nil {
conn.Close()
return
}
for _, song := range songs {
download(song, conn)
}
}
func download(song models.SongsModel, conn net.Conn) {
filePath, err := filepath.Abs(fmt.Sprintf("./songs/%s.flac", song.SongId))
if err != nil {
logger.ErrorLog(fmt.Sprintf("Errror: couldn't create filepath. Details: %v", err))
conn.Close()
return
}
file, err := os.Open(filePath)
defer file.Close()
if err != nil {
logger.ErrorLog(fmt.Sprintf("Errror: couldn't open file. Details: %v", err))
conn.Close()
return
}
read(file, conn)
}
func read(file *os.File, conn net.Conn) {
reader := bufio.NewReader(file)
buf := make([]byte, 15)
defer conn.Close()
for {
_, err := reader.Read(buf)
if err != nil && err == io.EOF {
logger.InfoLog(fmt.Sprintf("Details: %v", err))
fmt.Println()
return
}
conn.Write(buf)
}
}
CLIENT
main.go
func main() {
conn, _ := net.Dial("tcp", "127.0.0.1:6060")
var glMessage []byte
text := "eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6IjYzYzlhNmE1OWI3ZmQyNTQ2ZjA4ZWEyYSIsInVzZXJuYW1lIjoiMTIiLCJleHAiOjE2NzQyMTE5ODl9.aarSDhrFF1df3i2pIRyjNxTfSHKObqLU3kHJiPreredIhLNCzs7z7jMgRHQIcLaIvCOECN7bX0OaSvKdW7VKsQ\n"
fmt.Fprint(conn, text)
reader := bufio.NewReader(conn)
b := make([]byte, 15)
c := 0
for i, _ := reader.Read(b); int(i) != 0; i, _ = reader.Read(b) {
c += i
glMessage = append(glMessage, b...)
}
os.WriteFile("./test.flac", glMessage, 0644)
}
If you know what can be the problem, please tell me. I'd really appreciate it!
It looks like you're trying to send the music file over the network in 15 byte chunks, which is likely not enough to play the song on the client side.
You can try increasing the chunk size, for example, to 8192 bytes. To do this, replace buf := make([]byte, 15) with buf := make([]byte, 8192).
Also, it's better to write the received data directly to the file rather than storing it in memory. You can do this by creating a file and using os.Create to write the received data to it:
file, err := os.Create("./test.flac")
if err != nil {
fmt.Println("Error: couldn't create file")
return
}
defer file.Close()
for {
i, err := reader.Read(buf)
if err != nil && err == io.EOF {
break
}
file.Write(buf[:i])
}
I believe that this can solve the issue.

Cannot receive response packets from multicast server in golang

I setup a udp server listening multicast trafic, and create a client to sent test packet. When the server receives the package, it will send a response. Everything is ok, except the client cannot receive the packet from the server. Why?
package main
import (
"log"
"net"
"os"
"time"
"golang.org/x/net/ipv4"
)
func SetupMulticast(ifn, addr string) {
ifi, err := net.InterfaceByName(ifn)
if err != nil {
log.Fatal(err)
}
//server
c, err := net.ListenPacket("udp4", addr)
if err != nil {
log.Fatal(err)
}
log.Printf("listen ad:%s\n", addr)
p := ipv4.NewPacketConn(c)
gAddr, err2 := net.ResolveUDPAddr("udp4", addr)
if err2 != nil {
log.Fatal(err2)
}
if err := p.JoinGroup(ifi, gAddr); err != nil {
log.Fatal(err)
}
if err := p.SetControlMessage(ipv4.FlagDst|ipv4.FlagSrc, true); err != nil {
log.Fatal(err)
}
go func() {
b := make([]byte, 1500)
for {
n, cm, src, err := p.ReadFrom(b)
if err != nil {
break
}
log.Println("receive:", string(b[:n]), cm.Dst.IsMulticast(), cm.Dst)
if n2, err := p.WriteTo([]byte("world!"), nil, src); err != nil {
log.Printf("fail to write back:%v\n", err)
} else {
log.Printf("write back addr: %s length:%d\n", src, n2)
}
}
}()
//client
if conn, err2 := net.DialUDP("udp4", nil /*src*/, gAddr); err2 != nil {
log.Fatal(err)
} else {
go func() {
for {
log.Println("write hello...")
conn.Write([]byte("hello"))
time.Sleep(time.Second * 2)
}
}()
go func() {
b2 := make([]byte, 1500)
for {
n, err := conn.Read(b2)
if err != nil {
log.Panic(err)
}
log.Printf("sender received response:%s\n", string(b2[:n]))
}
}()
}
}
func main() {
If := os.Args[1] //ens33
Addr := os.Args[2] //224.0.0.248:1025
SetupMulticast(If, Addr)
for {
}
}
And the output:
2022/08/17 22:53:53 listen ad:224.0.0.248:1025
2022/08/17 22:53:53 write hello...
2022/08/17 22:53:53 receive: hello true 224.0.0.248
2022/08/17 22:53:53 write back addr: 192.168.19.131:43925 length:6
2022/08/17 22:53:55 write hello...
2022/08/17 22:53:55 receive: hello true 224.0.0.248
2022/08/17 22:53:55 write back addr: 192.168.19.131:43925 length:6
From the logs, there are no any "sender received response" record. I don't know why?

Write multiple time on TCP conn

I have a TCP server server :
l, err := net.Listen("tcp", "localhost:"+strconv.Itoa(tcpPort))
The server listens to incoming client requests as is :
for {
c, err := l.Accept()
if err != nil {
log.Fatal(err)
}
b := make([]byte, 1024)
c.Read(b)
fmt.Println(string(b)) // "Hello"
}
I have a client :
conn, err := net.Dial("tcp", address)
Now I I write once with conn.Write([]byte("Hello"))
The server catches Hello. But if I have these :
_, err := conn.Write([]byte("Hello"))
if err != nil {
log.Fatal(err)
}
_, err = conn.Write([]byte("World"))
if err != nil {
log.Fatal(err)
}
Then I will get Hello, but not World.
How can I write multiple time on the same connection ?
Full function below
func main() {
l, err := net.Listen("tcp", "localhost:1234")
if err != nil {
log.Fatal(err)
}
defer l.Close()
go func() {
for {
c, err := l.Accept()
if err != nil {
log.Fatal(err)
}
b := make([]byte, 1024)
c.Read(b)
fmt.Println(string(b))
}
}()
conn, err := net.Dial("tcp", "localhost:1234")
if err != nil {
log.Fatal(err)
}
_, err = conn.Write([]byte("hello"))
if err != nil {
log.Fatal(err)
}
_, err = conn.Write([]byte("world"))
if err != nil {
log.Fatal(err)
}
}
Your problem is in the server code, the one that receives data. It only reads once from the tcp stream. If you want it to read "world", replace the single read operation by a loop:
go func() {
for {
c, err := l.Accept()
if err != nil {
log.Fatal(err)
}
for {
b := make([]byte, 1024)
c.Read(b)
fmt.Println(string(b))
}
}
}()
Your problem is not with conn.Write but with reading from connection. Now you read just once from every new opened connection accepted by l.Accept(). Solution is to read repeatedly.
Your code is also limited to handle just one connection. And do not forget to check error from c.Read(b) to know when stop listen on this connection.
go func() {
for {
c, err := l.Accept()
if err != nil {
log.Fatal(err)
}
go func(conn net.Conn) {
for {
b := make([]byte, 1024)
_, err := conn.Read(b)
if err != nil {
if err != io.EOF {
fmt.Println("read error:", err)
}
break
}
fmt.Println(string(b))
}
fmt.Println("Stopping handle connection")
}(c)
}
}()

Golang io.copy not copying entire data

I have small code which reads 100 MB file from Google cloud storage and then return output.
Code works fine for 1MB file but fails for 100 mb file.
Below is the code which is not working
rc, err := client.Bucket("mabucket").Object(gcpurl).NewReader(ctx)
if err != nil {
fmt.Println(err)
return
}
defer rc.Close()
w.Header().Set("Content-Length", strconv.Itoa(int(rc.Size())))
w.Header().Set("Cache-Control", "max-age=2592000")
w.Header().Set("Content-Type", rc.ContentType())
spew.Dump(rc.ContentType())
if rc.ContentType() == "audio/wav" || rc.ContentType() == "audio/wave" {
w.Header().Set("Accept-Ranges", "bytes")
tilrange := rc.Size() - 1
newRangeString := "bytes 0-" + strconv.Itoa(int(tilrange)) + "/" + strconv.Itoa(int(rc.Size()))
w.Header().Set("Content-Range", newRangeString)
w.WriteHeader(206)
}
//spew.Dump(rc.Attrs)
io.Copy(w, rc)
I have written another code which is able to download same file and create a local file of 100 mb.
this time I am using ioutil.ReadAll. what can be problem with io.copy when receiving large date from GCP?
func main() {
ctx := context.Background()
client, _ := storage.NewClient(ctx)
data, err := downloadFile(client, "mabucket", "606ff2b71a916907409a953f/606ff2ed1a916907409a9540/60a38a967b291f7b44488824/123/audio/210415164000M29713363.wav")
//210415164000M29713363.wav
if err != nil {
log.Fatalf("Cannot read object: %v", err)
}
fmt.Printf("Object contents: %d\n", len(data))
f, err := os.Create("a.wav")
check(err)
defer f.Close()
n2, err := f.Write(data)
check(err)
fmt.Printf("wrote %d bytes\n", n2)
}
// downloadFile downloads an object.
func downloadFile(client *storage.Client, bucket, object string) ([]byte, error) {
// [START download_file]
ctx := context.Background()
ctx, cancel := context.WithTimeout(ctx, time.Second*50)
defer cancel()
rc, err := client.Bucket(bucket).Object(object).NewReader(ctx)
if err != nil {
return nil, err
}
defer rc.Close()
data, err := ioutil.ReadAll(rc)
if err != nil {
return nil, err
}
return data, nil
// [END download_file]
}
io.copy was trying to copy the details but OS was not allowing it. it was throwing error as (*net.OpError)(0xc0003302d0)(write tcp [::1]:80->[::1]:63014: wsasend: An established connection was aborted by the software in your host machine

Golang broken pipe

I have a code segment that is reading from a TCP connection and after the first few connections, the server is outputing broken pipe however no error is occurring in my go code. The server sending the messages is at its core the coda hale metrics library, more specifically the PickledGraphite class.
here is the Go code that is reading:
func handleConn(conn net.Conn, id int) {
fmt.Println("handleConn")
defer conn.Close()
buf := make([]byte, 0, 10240)
tmp := make([]byte, 256)
fmt.Printf("%v Reading...\n", id)
for {
n, err := conn.Read(tmp)
fmt.Printf("%v Read %v\n", id, n)
if err != nil {
fmt.Printf("%v Got err: %v\n", id, err)
if err != io.EOF {
fmt.Printf("%v read error: %v\n", id, err)
}
buf = append(buf, tmp[:n]...)
break
}
buf = append(buf, tmp[:n]...)
}
fmt.Printf("%v Done Reading\n", id)
// Do stuff with buf
}
func main() {
ln, err := net.Listen("tcp", ":5555")
if err != nil {
fmt.Println(err)
os.Exit(-1)
}
id := 1
for {
fmt.Println("getting connection\n")
conn, err := ln.Accept()
if err != nil {
fmt.Println(err)
break
}
conn.SetReadDeadline(time.Now().Add(20 * time.Second))
fmt.Println("Got connection")
go handleConn(conn, id)
id = id + 1
fmt.Println("sent handleConn\n")
}
}
My code is still running, I can still execute nc commands and see my code receive it, so I am not sure how I am losing the connection.
If I remove the conn.SetReadDeadline() line then what happens is my code no longer receives a EOF after the first message.
Thanks in advance

Resources