Read whole data with Golang net.Conn.Read - go

So I'm building a network app in Go and I've seen that Conn.Read reads into a limited byte array, which I had created with make([]byte, 2048) and now the problem is that I don't know the exact length of the content, so it could be too much or not enough.
My question is how can I just read the exact amount of data. I think I have to use bufio, but I'm not sure.

It highly depends on what you're trying to do, and what kind of data you're expecting, for example if you just want to read until the EOF you could use something like this:
func main() {
conn, err := net.Dial("tcp", "google.com:80")
if err != nil {
fmt.Println("dial error:", err)
return
}
defer conn.Close()
fmt.Fprintf(conn, "GET / HTTP/1.0\r\n\r\n")
buf := make([]byte, 0, 4096) // big buffer
tmp := make([]byte, 256) // using small tmo buffer for demonstrating
for {
n, err := conn.Read(tmp)
if err != nil {
if err != io.EOF {
fmt.Println("read error:", err)
}
break
}
//fmt.Println("got", n, "bytes.")
buf = append(buf, tmp[:n]...)
}
fmt.Println("total size:", len(buf))
//fmt.Println(string(buf))
}
//edit: for completeness sake and #fabrizioM's great suggestion, which completely skipped my mind:
func main() {
conn, err := net.Dial("tcp", "google.com:80")
if err != nil {
fmt.Println("dial error:", err)
return
}
defer conn.Close()
fmt.Fprintf(conn, "GET / HTTP/1.0\r\n\r\n")
var buf bytes.Buffer
io.Copy(&buf, conn)
fmt.Println("total size:", buf.Len())
}

You can use the ioutil.ReadAll function:
import (
"fmt"
"io/ioutil"
"net"
)
func whois(domain, server string) ([]byte, error) {
conn, err := net.Dial("tcp", server+":43")
if err != nil {
return nil, err
}
defer conn.Close()
fmt.Fprintf(conn, "%s\r\n", domain)
return ioutil.ReadAll(conn)
}

You can read data something like this:
// import net/textproto
import ("net/textproto", ...)
....
reader := bufio.NewReader(Conn)
tp := textproto.NewReader(reader)
defer Conn.Close()
for {
// read one line (ended with \n or \r\n)
line, _ := tp.ReadLine()
// do something with data here, concat, handle and etc...
}
....

Related

Transfering file using tcp golang

I'm trying to make a music app that sends file through tcp protocol using go and microservice architecture. Now I'm creating a player service that should:
Get user token and get claims from it
Check is user exists using claims and user_service microservice
Get song from redis
Check is song exists using music_service
Read file by chunks and send it to client using tcp
Redis data looks like this:
{
"user_id": [{
"song_id": "<song_id>"
}]
}
But I faced with a small problem. My music files stored in a flac format and when I receive it on the client, my player doesn't play it. I don't really know what can be the problem. So here's my code:
SERVER
service_setup.go
//this function is called in main function
func setService() {
ln, err := net.Listen("tcp", config.TCPAddress)
if err != nil {
panic("couldn't start tcp server")
}
defer ln.Close()
for {
conn, err := ln.Accept()
if err != nil {
logger.ErrorLog(fmt.Sprintf("Error: couldn't accept connection. Details: %v", err))
return
}
service.DownloadSong(conn)
}
}
downloader_service.go
func DownloadSong(conn net.Conn) {
token, err := bufio.NewReader(conn).ReadString('\n')
if err != nil {
logger.ErrorLog(fmt.Sprintf("Error: couldn't get token. Details: %v", token))
conn.Close()
return
}
claims, err := jwt_funcs.DecodeJwt(token)
if err != nil {
conn.Close()
return
}
songs, err := redis_repo.Get(claims.Id)
if err != nil {
conn.Close()
return
}
for _, song := range songs {
download(song, conn)
}
}
func download(song models.SongsModel, conn net.Conn) {
filePath, err := filepath.Abs(fmt.Sprintf("./songs/%s.flac", song.SongId))
if err != nil {
logger.ErrorLog(fmt.Sprintf("Errror: couldn't create filepath. Details: %v", err))
conn.Close()
return
}
file, err := os.Open(filePath)
defer file.Close()
if err != nil {
logger.ErrorLog(fmt.Sprintf("Errror: couldn't open file. Details: %v", err))
conn.Close()
return
}
read(file, conn)
}
func read(file *os.File, conn net.Conn) {
reader := bufio.NewReader(file)
buf := make([]byte, 15)
defer conn.Close()
for {
_, err := reader.Read(buf)
if err != nil && err == io.EOF {
logger.InfoLog(fmt.Sprintf("Details: %v", err))
fmt.Println()
return
}
conn.Write(buf)
}
}
CLIENT
main.go
func main() {
conn, _ := net.Dial("tcp", "127.0.0.1:6060")
var glMessage []byte
text := "eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6IjYzYzlhNmE1OWI3ZmQyNTQ2ZjA4ZWEyYSIsInVzZXJuYW1lIjoiMTIiLCJleHAiOjE2NzQyMTE5ODl9.aarSDhrFF1df3i2pIRyjNxTfSHKObqLU3kHJiPreredIhLNCzs7z7jMgRHQIcLaIvCOECN7bX0OaSvKdW7VKsQ\n"
fmt.Fprint(conn, text)
reader := bufio.NewReader(conn)
b := make([]byte, 15)
c := 0
for i, _ := reader.Read(b); int(i) != 0; i, _ = reader.Read(b) {
c += i
glMessage = append(glMessage, b...)
}
os.WriteFile("./test.flac", glMessage, 0644)
}
If you know what can be the problem, please tell me. I'd really appreciate it!
It looks like you're trying to send the music file over the network in 15 byte chunks, which is likely not enough to play the song on the client side.
You can try increasing the chunk size, for example, to 8192 bytes. To do this, replace buf := make([]byte, 15) with buf := make([]byte, 8192).
Also, it's better to write the received data directly to the file rather than storing it in memory. You can do this by creating a file and using os.Create to write the received data to it:
file, err := os.Create("./test.flac")
if err != nil {
fmt.Println("Error: couldn't create file")
return
}
defer file.Close()
for {
i, err := reader.Read(buf)
if err != nil && err == io.EOF {
break
}
file.Write(buf[:i])
}
I believe that this can solve the issue.

GO programming, blocking read function on a reader event

I am a beginner in Go programming and I am confused about a problem about bufio readers.
I'm programming a kind of chat client who must display and send our messages in live time. But messages that I receive are not displayed until I press enter in my terminal.
After few tests, it seems to be my "inputListener()" function being the problem because, if I put it after reading messages from server, messages from server are displaying first.
I supposed that the Read function may blocked my loop until it get a '\n' or something like that.
Here is my code:
package main
import "os"
import "strconv"
import "net"
import "bufio"
/*Recovery our input message into a buffer*/
func inputListener()([] byte){
buf := make([]byte, 512)
readerInput := bufio.NewReader(os.Stdin)
_, err := readerInput.Read(buf)
if err != nil{
panic("Error reading input.")
os.Exit(0)
}
return buf
}
func main(){
if len(os.Args) != 3{
println("Usage: ",os.Args[0], " <host> <port>\n")
os.Exit(0)
}
//Recovery the port.
port, err := strconv.Atoi(os.Args[2])
if err != nil{
panic("Error during the port recovery\n")
os.Exit(0)
}
println(port)
/*Join the adresse*/
addr := os.Args[1] + ":" + strconv.Itoa(port)
println(addr)
/* sources -- https://golang.org/pkg/net/ */
conn, err := net.Dial("tcp", addr)
if err != nil{
panic("Error connecting " + addr)
os.Exit(0)
}
buf := make([]byte, 512)
t := make([]byte, 512)
for {
/*Receive data from server*/
size, err := conn.Read(buf)
if err != nil {
panic("Error reading output.")
os.Exit(0)
}
if size >= 0{
print(string(buf[0:size]))
}
/*Data we need to send*/
t = inputListener()
if len(t) >= 0{
conn.Write(t)
}
}
conn.Close()
}
I need to press enter per messages received :/
Thank you in advance for your answers !
You be try something like this:
package main
import (
"bufio"
"io"
"net"
"os"
"strconv"
)
/*Recovery our input message into a buffer*/
func inputListener() []byte {
buf := make([]byte, 512)
readerInput := bufio.NewReader(os.Stdin)
_, err := readerInput.Read(buf)
if err != nil {
panic("Error reading input.")
}
return buf
}
func main() {
if len(os.Args) != 3 {
println("Usage: ", os.Args[0], " <host> <port>\n")
os.Exit(0)
}
//Recovery the port.
port, err := strconv.Atoi(os.Args[2])
if err != nil {
panic("Error during the port recovery\n")
}
println(port)
/*Join the adresse*/
addr := os.Args[1] + ":" + strconv.Itoa(port)
println(addr)
/* sources -- https://golang.org/pkg/net/ */
conn, err := net.Dial("tcp", addr)
if err != nil {
panic("Error connecting " + addr)
}
defer conn.Close()
go io.Copy(os.Stdout, conn)
r := bufio.NewReader(os.Stdin)
for {
p, err := r.ReadSlice('\n')
if err != nil {
panic("Error reading output.")
}
conn.Write(p)
}
}

Client stuck when trying to read with io.CopyN() in golang

I am trying to make TCP server for transferring files. I am suing io.CopyN for reading and writing. From server side, I am sending files to client so from server side, it sends perfectly all bytes but Client side after reading a couple of 1000000 bytes it stuck. sometimes it works fine and sometimes it gets stuck. I am using 300 MB pdf to test. Any help, code, and output is like below.
server
package main
import (
"fmt"
"io"
"log"
"net"
"os"
"strconv"
"strings"
)
func main() {
ls, err := net.Listen("tcp", ":1234")
errFunc(err)
defer ls.Close()
conn, _ := ls.Accept()
defer conn.Close()
for {
file, err := os.Open(strings.TrimSpace("./" + "Mag" + ".pdf"))
errFunc(err)
defer file.Close()
fileInfo, err := file.Stat()
errFunc(err)
size := fileInfo.Size()
numberOfTime := size / 1000000
leftByte := size - numberOfTime*1000000
numberOfTimeString := strconv.Itoa(int(numberOfTime))
leftByteString := strconv.Itoa(int(leftByte))
fmt.Println("1000000 times : ", numberOfTimeString)
fmt.Println("Left Bytes : ", leftByteString)
_, err = fmt.Fprintf(conn, numberOfTimeString+"\n")
errFunc(err)
_, err = fmt.Fprintf(conn, leftByteString+"\n")
errFunc(err)
fileWriter := io.Writer(conn)
for i := 0; i < int(numberOfTime); i++ {
n, err := io.CopyN(conn, file, 1000000)
if i >= 30 {
fmt.Println(err, n)
}
}
n, err := io.CopyN(fileWriter, file, leftByte+1)
if err == io.EOF {
fmt.Println(err, n)
}
fmt.Printf("Succefully bytes sent : %v \n\n\n\n\n", n)
file.Close()
}
}
func errFunc(err error) {
if err != nil {
log.Fatal(err)
}
}
client
package main
import (
"bufio"
"fmt"
"io"
"net"
"os"
"os/signal"
"strconv"
"strings"
"syscall"
)
func main() {
c := make(chan os.Signal, 15)
signal.Notify(c, syscall.SIGINT)
go func() {
for {
s := <-c
switch s {
case syscall.SIGINT:
os.Exit(1)
}
}
}()
conn, _ := net.Dial("tcp", ":1234")
defer conn.Close()
connReadWrite := bufio.NewReader(io.Reader(conn))
var i int
var filename string
for {
i++
nu := strconv.Itoa(i)
filename = "image" + nu + ".pdf"
file, err := os.Create(filename)
defer file.Close()
numberOfTimeString, err := connReadWrite.ReadString('\n')
if err != nil {
fmt.Println(err)
}
println("1000000 times :", numberOfTimeString)
numberOfTimeString = strings.TrimSuffix(numberOfTimeString, "\n")
numberOfTime, err := strconv.Atoi(numberOfTimeString)
if err != nil {
fmt.Println(err)
}
leftByteString, err := connReadWrite.ReadString('\n')
if err != nil {
println(err)
}
println("Left Bytes :", leftByteString)
leftByteString = strings.TrimSuffix(leftByteString, "\n")
leftByte, err := strconv.Atoi(leftByteString)
if err != nil {
panic(err)
}
fmt.Println("After convert in Num :", numberOfTime, leftByte)
newFileWriter := io.Writer(file)
newFileReader := io.Reader(conn)
for i := 0; i < numberOfTime; i++ {
n, err := io.CopyN(newFileWriter, newFileReader, 1000000)
if i >= 30 {
errFun(err, n)
}
}
n, err := io.CopyN(newFileWriter, newFileReader, int64(leftByte))
errFun(err, n)
fmt.Printf("sucessfully Transfered ---> \n\n\n\n\n\n")
}
}
func errFun(err error, n int64) {
if err == io.EOF {
fmt.Println("End of file : ", n)
return
} else if n == 0 {
fmt.Println("n is : ", n)
return
} else if err != nil {
fmt.Println(err)
return
}
fmt.Println(err, " : ", n)
}
input/output
from server side first we are sending number of bytes it need to readand then client side it gets a number of bytes it needs to read and then I am sending the file and then it read. In the picture, I was able to send one-time second time it got stuck sometimes it stuck first time too.I am able to send number of byte from server side second time too but as you can see it don't read that numeber, it read something "%PDF..." and it even don't print "100000 times : " correctly it prints "%???00 times :" I just don’t understand this
enter image description here
I believe the issue is that you're using a bytes.Buffer in the client:
connReadWrite := bufio.NewReader(io.Reader(conn))
But you aren't using it later with the CopyN:
newFileWriter := io.Writer(file)
newFileReader := io.Reader(conn)
for i := 0; i < numberOfTime; i++ {
_, err := io.CopyN(newFileWriter, newFileReader, 1000000)
if err != nil {
log.Fatalln(err)
}
}
Using:
newFileWriter := io.Writer(file)
for i := 0; i < numberOfTime; i++ {
_, err := io.CopyN(file, connReadWrite, 1000000)
if err != nil {
log.Fatalln(err)
}
}
May fix it.
If you have control over the protocol you are using to send the file, I recommend doing something simpler. For example using the big-endian int64 length prefix.
Send:
func sendFile(name string, conn net.Conn) error {
f, err := os.Open(name)
if err != nil {
return err
}
defer f.Close()
fi, err := f.Stat()
if err != nil {
return err
}
sz := fi.Size()
buf := bufio.NewWriter(conn)
err = binary.Write(buf, binary.BigEndian, sz)
if err != nil {
return err
}
_, err = io.CopyN(buf, f, sz)
if err != nil {
return err
}
return buf.Flush()
}
Receive:
func recvFile(name string, conn net.Conn) error {
f, err := os.Create(name)
if err != nil {
return err
}
defer f.Close()
buf := bufio.NewReader(conn)
var sz int64
err = binary.Read(buf, binary.BigEndian, &sz)
if err != nil {
return err
}
_, err = io.CopyN(f, buf, sz)
if err != nil {
return err
}
return nil
}

How to pipe an HTTP response to a file in Go?

How do I convert the below code to use streams/pipes so that I don't need to read the full content into memory?
Something like:
http.Get("http://example.com/").Pipe("./data.txt")
package main
import ("net/http";"io/ioutil")
func main() {
resp, err := http.Get("http://example.com/")
check(err)
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
check(err)
err = ioutil.WriteFile("./data.txt", body, 0666)
check(err)
}
func check(e error) {
if e != nil {
panic(e)
}
}
How about io.Copy()? Its documentation can be found at: http://golang.org/pkg/io/#Copy
It's pretty simple, though. Give it an io.Reader and an io.Writer and it copies the data over, one small chunk at a time (e.g. not all in memory at once).
So you might try writing something like:
func main() {
resp, err := http.Get("...")
check(err)
defer resp.Body.Close()
out, err := os.Create("filename.ext")
if err != nil {
// panic?
}
defer out.Close()
io.Copy(out, resp.Body)
}
I haven't tested the above; I just hacked it together quickly from your above example, but it should be close if not on the money.
Another option is File.ReadFrom:
package main
import (
"net/http"
"os"
)
func main() {
r, e := http.Get("http://speedtest.lax.hivelocity.net")
if e != nil {
panic(e)
}
defer r.Body.Close()
f, e := os.Create("index.html")
if e != nil {
panic(e)
}
defer f.Close()
f.ReadFrom(r.Body)
}

How can I efficiently download a large file using Go?

Is there a way to download a large file using Go that will store the content directly into a file instead of storing it all in memory before writing it to a file? Because the file is so big, storing it all in memory before writing it to a file is going to use up all the memory.
I'll assume you mean download via http (error checks omitted for brevity):
import ("net/http"; "io"; "os")
...
out, err := os.Create("output.txt")
defer out.Close()
...
resp, err := http.Get("http://example.com/")
defer resp.Body.Close()
...
n, err := io.Copy(out, resp.Body)
The http.Response's Body is a Reader, so you can use any functions that take a Reader, to, e.g. read a chunk at a time rather than all at once. In this specific case, io.Copy() does the gruntwork for you.
A more descriptive version of Steve M's answer.
import (
"os"
"net/http"
"io"
)
func downloadFile(filepath string, url string) (err error) {
// Create the file
out, err := os.Create(filepath)
if err != nil {
return err
}
defer out.Close()
// Get the data
resp, err := http.Get(url)
if err != nil {
return err
}
defer resp.Body.Close()
// Check server response
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("bad status: %s", resp.Status)
}
// Writer the body to file
_, err = io.Copy(out, resp.Body)
if err != nil {
return err
}
return nil
}
The answer selected above using io.Copy is exactly what you need, but if you are interested in additional features like resuming broken downloads, auto-naming files, checksum validation or monitoring progress of multiple downloads, checkout the grab package.
Here is a sample. https://github.com/thbar/golang-playground/blob/master/download-files.go
Also I give u some codes might help you.
code:
func HTTPDownload(uri string) ([]byte, error) {
fmt.Printf("HTTPDownload From: %s.\n", uri)
res, err := http.Get(uri)
if err != nil {
log.Fatal(err)
}
defer res.Body.Close()
d, err := ioutil.ReadAll(res.Body)
if err != nil {
log.Fatal(err)
}
fmt.Printf("ReadFile: Size of download: %d\n", len(d))
return d, err
}
func WriteFile(dst string, d []byte) error {
fmt.Printf("WriteFile: Size of download: %d\n", len(d))
err := ioutil.WriteFile(dst, d, 0444)
if err != nil {
log.Fatal(err)
}
return err
}
func DownloadToFile(uri string, dst string) {
fmt.Printf("DownloadToFile From: %s.\n", uri)
if d, err := HTTPDownload(uri); err == nil {
fmt.Printf("downloaded %s.\n", uri)
if WriteFile(dst, d) == nil {
fmt.Printf("saved %s as %s\n", uri, dst)
}
}
}

Resources