I've just recently started trying to learn Go and I'm trying to write a small server/client application for sending a csv file, from a server to a client. I'm running into an invalid type error when trying to encode a struct into BigEndian binary. My struct seems to already be in a binary format, I'm unsure why I get the following error:
Error writing binary buffer to big endian binary.Write: invalid type *main.DataPack
Also, I'd like to keep the TCP connection open after sending the file, that's why I'm not using io.Copy.
Currently I'm triggering the handling of the file upload through by sending a '\x00' byte:
// Server
package main
import (
"path/filepath"
"fmt"
"io/ioutil"
"net"
"os"
"encoding/binary"
"bytes"
)
type DataPack struct {
Length int64
Contents []byte
}
func main() {
absPath, _ := filepath.Abs("./progs.csv")
data, _ := ioutil.ReadFile(absPath)
fmt.Println("%s",data)
tel, err := net.Listen("tcp", "0.0.0.0:23")
if err != nil {
fmt.Println(err)
return
}
for {
conn, err := tel.Accept()
if err != nil {
break
}
fmt.Println("above filehandler")
go fileHandler(conn)
}
}
func fileHandler(conn net.Conn) {
buf := make([]byte, 0)
buf = append(buf, '\x00')
conn.Write(buf)
absPath, _ := filepath.Abs("./progs.csv")
file, err := os.Open(absPath)
defer file.Close()
fi, err := file.Stat()
if err != nil {
fmt.Println("Fatal error reading file: ", err)
}
fmt.Println("This is the length of the file: ", fi.Size())
data := &DataPack{Length: fi.Size()} // , Contents: }
data.Contents = make([]byte, data.Length)
n, err := file.Read(data.Contents)
if err != nil {
fmt.Println("error reading contents into struct: ", err)
}
fmt.Println("tried to read file contents: ", n)
fmt.Println("DataPack: %+v", data)
buf1 := new(bytes.Buffer)
err = binary.Write(buf1, binary.BigEndian, &data)
if err != nil {
fmt.Println("Error writing binary buffer to big endian ", err)
}
conn.Write(buf1.Bytes())
}
Here is the client
package main
import (
"log"
"fmt"
"net"
"strings"
"strconv"
"bufio"
)
const (
host = "127.0.0.1"
port = 23
)
func main() {
addr := strings.Join([]string{host, strconv.Itoa(port)}, ":")
client := NewClient()
var err error
client.socket, err = net.Dial("tcp", addr)
if err != nil {
log.Println("error setting up socket: %s", err)
}
for {
m := bufio.NewReader(client.socket)
b, err := m.ReadByte()
if err != nil {
fmt.Println("here is the error: ", err)
}
if b == '\x00'{
fmt.Println("about to receive a file!!!")
b, _ := m.ReadByte()
fmt.Println("just got another byte ", b )
}
}
log.Printf("Over")
}
why you get the error
Visit https://golang.org/pkg/encoding/binary/#Write
Write writes the binary representation of data into w. Data must be a fixed-size value or a slice of fixed-size values, or a pointer to such data.
type DataPack struct {
Length int64
Contents []byte
}
Contents isn't a fixed-size value, so you got the invalid type error.
how to solve it
go binary
json
others
Related
I want to read gzip compressed file from blob storage. I know an URL of storage, offset and length on needed file. So the most simple way to download the file id to use http.client & gzip.Reader
My code is
package main
import (
"bytes"
"compress/gzip"
"fmt"
"io/ioutil"
"net/http"
"os"
)
func main() {
req, _ := http.NewRequest("GET", "https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2021-10/segments/1614178385529.97/warc/CC-MAIN-20210308205020-20210308235020-00200.warc.gz", nil)
req.Header.Add("Range", "bytes=842089698-842207846")
var client http.Client
resp, err := client.Do(req)
if err != nil {
fmt.Println(err)
os.Exit(0)
}
// for key, value := range resp.Header {
// fmt.Printf("%s: %s\n", key, value)
// }
// fmt.Printf("length: %v\n", resp.ContentLength)
option := 2
if option == 1 {
// --- Option 1 ----
reader, _ := gzip.NewReader(resp.Body)
body, err := ioutil.ReadAll(reader)
if err != nil {
fmt.Println(err)
os.Exit(0)
}
fmt.Println(string(body))
} else {
//--- Option 2 ----
body, _ := ioutil.ReadAll(resp.Body)
gr, _ := gzip.NewReader(bytes.NewBuffer(body))
body, _ = ioutil.ReadAll(gr)
fmt.Println(string(body))
}
}
If I set option variable to 2 - everything works fine, but if use option #1 programm ends with a error "unexpected EOF". Why does it happen?
I am trying to create a tcp server/client connection in golang. I am sending a message length so the server will accept multiple messages until the message length is reached. When I try to read the message header to determine the message length, there is no output. The code seems to gets stuck in the server's tcpreader function's for loop when "conn.read(b)" is called. What might be causing this failure?
My server code:
package main
import (
"bufio"
"bytes"
"encoding/binary"
"fmt"
"log"
"net"
"strings"
)
type hdr struct {
magicNum uint64
msgLen uint32
msgType uint16
padding uint16
}
func tcpReader(conn net.Conn) {
foundLength := false
var hdr struct {
magicNum uint64
msgLen uint32
msgType uint16
padding uint16
}
hdrSize := 16
fmt.Println("tcpReader() header size %v", hdrSize)
//localBuffer := new(bytes.Buffer)
defer conn.Close()
for {
if !foundLength {
fmt.Println("tcpReader() Getting the lenght of the message")
var b = make([]byte, hdrSize)
// where the code seems to get stuck
read, err := conn.Read(b)
if err != nil {
fmt.Println("Failed to read header size %v", hdrSize)
fmt.Println(err)
continue
}
fmt.Println("tcpReader() read header bytes %v %v x", read, string(b))
if read != hdrSize { //add logic check magic number etc
fmt.Println("invalid header")
continue
}
err = binary.Read(bytes.NewReader(b[:16]), binary.BigEndian, &hdr)
if err != nil {
log.Fatal(err)
}
fmt.Println("tcpReader() read header bytes %v", hdr.msgLen)
foundLength = true
// messageLength = int(binary.BigEndian.Uint64(b))
} else {
var message = make([]byte, hdr.msgLen)
read, err := conn.Read(message)
if err != nil {
fmt.Println(err)
continue
}
fmt.Println("Received bytes: %v", read)
if read != int(hdr.msgLen) {
//Add logic to append the read message and update the readSofarBytes etc
fmt.Println("invalid data")
continue
}
fmt.Println("Received:", string(message))
foundLength = false
}
}
}
func main() {
fmt.Println("Launching server...")
fmt.Println("Listen on port")
/* addr, err := net.ResolveTCPAddr("ip4", "127.0.0.1:8081")
if err != nil {
panic(err)
}*/
ln, err := net.Listen("tcp", "127.0.0.1:8081")
if err != nil {
log.Fatal(err)
}
defer ln.Close()
fmt.Println("Accept connection on port")
for {
conn, err := ln.Accept()
if err != nil {
log.Fatal(err)
}
fmt.Println("Calling handleConnection")
//go handleConnection(conn)
go tcpReader(conn)
}
}
My client code:
package main
import (
"bufio"
"bytes"
"encoding/binary"
"fmt"
"net"
)
type hdr struct {
magicNum uint64
msgLen int
msgType uint16
padding uint16
msg string
}
func main() {
var bin_buf bytes.Buffer
msg := hdr{magicNum: 123456789,
msgLen: 60,
msgType: 2,
padding: 0,
msg: "Hello world this is tcp message with big header"}
msg.msgLen = int(len(msg.msg))
fmt.Printf("message length %v\n", msg.msgLen)
binary.Write(&bin_buf, binary.BigEndian, msg)
addr, _ := net.ResolveTCPAddr("tcp", ":8081")
conn, err := net.DialTCP("tcp", nil, addr)
if err != nil {
panic(err.Error())
}
// conn.Write(bin_buf.Bytes())
conn.Write(bin_buf.Bytes())
//conn.Write([]byte("Hello from Clinet\n"))
message, _ := bufio.NewReader(conn).ReadString('\n')
fmt.Print(message)
conn.Close()
}
It is because you are not writing anything from the client side at all. You should never ignore error handling in golang.Change these lines and you'll see where the error is:
err := binary.Write(&bin_buf, binary.BigEndian, msg)
if err != nil {
panic(err.Error())
}
Client is writing 0 bytes to connection because there aren't any
panic: binary.Write: invalid type main.hdr
To complete Keser's answer, you can't use a struct whose fields have types without a fixed size. In this case your hdr struct has two fields that can't be written using binary.Write, the message field with type string and msgLen without specifying the int size. You will have to manually write the string message as it is up to the developer to figure out how to handle strings when writing it through a TCP connection. Additionally, all the fields of your struct are unexported, binary.Write will not be able to access them. Use an uppercase letter to export them.
If you try it, you'll know it.
func main() {
...
err := binary.Write(&bin_buf, binary.BigEndian, msg)
fmt.Println(err)
fmt.Println(string(bin_buf.Bytes()))
...
}
I'm studying networks, and I'm doing a tcp server with Go. One of the challenges I'm studying is to send binaries or strings by a socket client to a server, save the server response to a txt, and compare it to the original data that was sent.
The problem is that the binaries do not arrive completely on the server.
Server
package main
import (
"fmt"
"io"
"log"
"net"
)
func main() {
l, err := net.Listen("tcp", ":8000")
if nil != err {
log.Println(err)
}
defer l.Close()
for {
conn, err := l.Accept()
if nil != err {
log.Println(err)
continue
}
defer conn.Close()
go ConnHandler(conn)
}
}
func ConnHandler(conn net.Conn) {
recvBuf := make([]byte, 4096)
for {
n, err := conn.Read(recvBuf)
if nil != err {
if io.EOF == err {
log.Println(err)
return
}
log.Println(err)
return
}
if 0 < n {
data := recvBuf[:n]
fmt.Println(string(data))
}
}
}
Client
package main
import (
"fmt"
"log"
"net"
)
func main() {
conn, err := net.Dial("tcp", ":8000")
if nil != err {
log.Println(err)
}
var s string
fmt.Scanln(&s)
conn.Write([]byte(s))
conn.Close()
}
I'm generating the binaries using the command on linux:
head -c100000 /dev/urandom > binary_message.txt
I run the server:
./server > result.txt
And I send this data by the client using:
./client < binary_data.txt
In the end the file binary_data.txt have 98KB but the result .txt only has 0KB.
The problem is with scanning the binary from input. You didn't see it because the errors were ignored and not printed or otherwise handled. fmt.Scanln returns an error (so does the Write function). You should always check for possible errors happening.
I rewrote the client to load the file from disk itself as I don't think using stdin is a good fit for binary data.
package main
import (
"flag"
"io"
"log"
"net"
"os"
)
var fileName = flag.String("file", "", "file to send")
func main() {
flag.Parse()
conn, err := net.Dial("tcp", ":8000")
if nil != err {
log.Println(err)
}
defer conn.Close()
f, err := os.Open(*fileName)
if err != nil {
log.Println(err)
return
}
defer f.Close()
b := make([]byte, 1024)
for {
n, err := f.Read(b)
if err != nil {
if err == io.EOF {
log.Println("Done sending")
return
}
log.Println(err)
return
}
if _, err := conn.Write(b[:n]); err != nil {
log.Println(err)
return
}
}
}
You can use it with:
go run . -file=binary_message.txt
or if you have built the binary:
./client -file=binary_message.txt
I suggest you do the same for the server. Open a file for writing and write the binary data into that file. Use a flag to pass in the filename to write to. That will be cleaner than piping stdout to a file.
I am a beginner in Go programming and I am confused about a problem about bufio readers.
I'm programming a kind of chat client who must display and send our messages in live time. But messages that I receive are not displayed until I press enter in my terminal.
After few tests, it seems to be my "inputListener()" function being the problem because, if I put it after reading messages from server, messages from server are displaying first.
I supposed that the Read function may blocked my loop until it get a '\n' or something like that.
Here is my code:
package main
import "os"
import "strconv"
import "net"
import "bufio"
/*Recovery our input message into a buffer*/
func inputListener()([] byte){
buf := make([]byte, 512)
readerInput := bufio.NewReader(os.Stdin)
_, err := readerInput.Read(buf)
if err != nil{
panic("Error reading input.")
os.Exit(0)
}
return buf
}
func main(){
if len(os.Args) != 3{
println("Usage: ",os.Args[0], " <host> <port>\n")
os.Exit(0)
}
//Recovery the port.
port, err := strconv.Atoi(os.Args[2])
if err != nil{
panic("Error during the port recovery\n")
os.Exit(0)
}
println(port)
/*Join the adresse*/
addr := os.Args[1] + ":" + strconv.Itoa(port)
println(addr)
/* sources -- https://golang.org/pkg/net/ */
conn, err := net.Dial("tcp", addr)
if err != nil{
panic("Error connecting " + addr)
os.Exit(0)
}
buf := make([]byte, 512)
t := make([]byte, 512)
for {
/*Receive data from server*/
size, err := conn.Read(buf)
if err != nil {
panic("Error reading output.")
os.Exit(0)
}
if size >= 0{
print(string(buf[0:size]))
}
/*Data we need to send*/
t = inputListener()
if len(t) >= 0{
conn.Write(t)
}
}
conn.Close()
}
I need to press enter per messages received :/
Thank you in advance for your answers !
You be try something like this:
package main
import (
"bufio"
"io"
"net"
"os"
"strconv"
)
/*Recovery our input message into a buffer*/
func inputListener() []byte {
buf := make([]byte, 512)
readerInput := bufio.NewReader(os.Stdin)
_, err := readerInput.Read(buf)
if err != nil {
panic("Error reading input.")
}
return buf
}
func main() {
if len(os.Args) != 3 {
println("Usage: ", os.Args[0], " <host> <port>\n")
os.Exit(0)
}
//Recovery the port.
port, err := strconv.Atoi(os.Args[2])
if err != nil {
panic("Error during the port recovery\n")
}
println(port)
/*Join the adresse*/
addr := os.Args[1] + ":" + strconv.Itoa(port)
println(addr)
/* sources -- https://golang.org/pkg/net/ */
conn, err := net.Dial("tcp", addr)
if err != nil {
panic("Error connecting " + addr)
}
defer conn.Close()
go io.Copy(os.Stdout, conn)
r := bufio.NewReader(os.Stdin)
for {
p, err := r.ReadSlice('\n')
if err != nil {
panic("Error reading output.")
}
conn.Write(p)
}
}
This question already has answers here:
How can I read a whole file into a string variable
(7 answers)
Closed 4 years ago.
I'm trying to read "file.txt" and put the contents into a variable using Golang. Here is what I've tried...
package main
import (
"fmt"
"os"
"log"
)
func main() {
file, err := os.Open("file.txt")
if err != nil {
log.Fatal(err)
}
fmt.Print(file)
}
The file gets read successfully and the return from os.Open returns a type of *os.File
It depends on what you are trying to do.
file, err := os.Open("file.txt")
fmt.print(file)
The reason it outputs &{0xc082016240}, is because you are printing the pointer value of a file-descriptor (*os.File), not file-content. To obtain file-content, you may READ from a file-descriptor.
To read all file content(in bytes) to memory, ioutil.ReadAll
package main
import (
"fmt"
"io/ioutil"
"os"
"log"
)
func main() {
file, err := os.Open("file.txt")
if err != nil {
log.Fatal(err)
}
defer func() {
if err = file.Close(); err != nil {
log.Fatal(err)
}
}()
b, err := ioutil.ReadAll(file)
fmt.Print(b)
}
But sometimes, if the file size is big, it might be more memory-efficient to just read in chunks: buffer-size, hence you could use the implementation of io.Reader.Read from *os.File
func main() {
file, err := os.Open("file.txt")
if err != nil {
log.Fatal(err)
}
defer func() {
if err = file.Close(); err != nil {
log.Fatal(err)
}
}()
buf := make([]byte, 32*1024) // define your buffer size here.
for {
n, err := file.Read(buf)
if n > 0 {
fmt.Print(buf[:n]) // your read buffer.
}
if err == io.EOF {
break
}
if err != nil {
log.Printf("read %d bytes: %v", n, err)
break
}
}
}
Otherwise, you could also use the standard util package: bufio, try Scanner. A Scanner reads your file in tokens: separator.
By default, scanner advances the token by newline (of course you can customise how scanner should tokenise your file, learn from here the bufio test).
package main
import (
"fmt"
"os"
"log"
"bufio"
)
func main() {
file, err := os.Open("file.txt")
if err != nil {
log.Fatal(err)
}
defer func() {
if err = file.Close(); err != nil {
log.Fatal(err)
}
}()
scanner := bufio.NewScanner(file)
for scanner.Scan() { // internally, it advances token based on sperator
fmt.Println(scanner.Text()) // token in unicode-char
fmt.Println(scanner.Bytes()) // token in bytes
}
}
Lastly, I would also like to reference you to this awesome site: go-lang file cheatsheet. It encompassed pretty much everything related to working with files in go-lang, hope you'll find it useful.