TCP fixed size message framing method in Golang - go

Question

I'm not able to understand how message framing using fixed size prefix length header works.
It's said that a fixed size byte array is going to contain the length for message to be sent. But how would you define a fixed size byte array, specifically in Golang.
Say this is my message:
Hello
Its length is 5.
So if I want to send this through a tcp stream, to make sure I receive all the message on the other end I'd have to tell it how many bytes it should read.
A simple header would be length:message:
5:Hello // [53 58 104 101 108 108 111]
But if message length grows 10x each time, there are going to be more bytes. So header size is dynamic that way.
36:Hello, this is just a dumb question. // [51 54 58 72 101 108 108 111 44 32 116 104 105 115 32 105 115 32 106 117 115 116 32 97 32 100 117 109 98 32 113 117 101 115 116 105 111 110 46]
So here 36 takes 2 bytes.
One approach I come to think of is to consider a maximum message length for the protocol. Say 10KB = 10240 bytes. Then add leading 0's to message length. This way I'm sure that I'm going to have a fixed 5 bytes header.
Would this work for all cases?
If yes, what if I have a message more than 10KBs, should I split it into 2 messages?
If not, what are other solutions?
I want to implement the solutions in Golang.
UPDATE 1:
I read about Endians, although I wasn't able to understand what they do that causes fixed length bytes. But I found an example in python and tried to write it in go this way:
Client:
const maxLengthBytes = 8
conn, err := net.Dial("tcp", "127.0.0.1:9999")
if err != nil {
fmt.Println(err)
return
}
message := "Hello, this is just a dumb question"
bs := make([]byte, maxLengthBytes)
binary.LittleEndian.PutUint64(bs, uint64(len(text)))
bytes := append(bs, []byte(text)...)
conn.Write(bytes)
Server:
listener, err := net.ListenTCP("tcp", &net.TCPAddr{Port: 9999})
if err != nil {
fmt.Println(err)
return
}
for {
tcp, err := listener.AcceptTCP()
if err != nil {
fmt.Println(err)
continue
}
go Reader(tcp)
}
func Reader(conn *net.TCPConn) {
foundLength := false
messageLength := 0
for {
if !foundLength {
var b = make([]byte, maxLengthBytes)
read, err := conn.Read(b)
if err != nil {
fmt.Println(err)
continue
}
if read != 8 {
fmt.Println("invalid header")
continue
}
foundLength = true
messageLength = int(binary.LittleEndian.Uint64(b))
} else {
var message = make([]byte, messageLength)
read, err := conn.Read(message)
if err != nil {
fmt.Println(err)
continue
}
if read != messageLength {
fmt.Println("invalid data")
continue
}
fmt.Println("Received:", string(message))
foundLength = false
messageLength = 0
}
}
}

Answer 1

Please refer to my answer in this post
TCP client for Android: text is not received in full
Basically, you have to define how the data stored/formatted.
For example:
We store prefix length as int32 (4 bytes) with little endian. It's different from yours.
With your solution, the length is a string, it's hard to fix the length.
For your solution, you have to use fixed length string. For example: 10 characters, and add leading zero.
For your questions.
It doesn't work for all cases with just prefix length. It has its limitation, for example if we use int32 as the prefix length, the length of message must be less than Integer32.max, right?
Yes, we have to split or even combine (please refer my explanation in above link).
We have many ways to deal with length limitation if it's our concern (actually, almost application protocols has it maximum request size).
You could use one more bit to indicate, whether or not the message exceeds max length to resolve it, right?