I am making a go program where I need to write a gob to a file. I used the .String() method to convert the gob to a string.
var network bytes.Buffer
encoder := gob.NewEncoder(&network)
_ = encoder.Encode(valueToEncode)
gobString := network.String()
then I will write the gob to a file, and later I will retrieve it and send it to this program:
var filebytes = []byte(file) //i think that these two lines are the issue
network := bytes.NewBuffer(filebytes)
decoder := gob.NewDecoder(network)
var decoded interface{}
_ := decoder.Decode(&decoded)
but when i run this, it gives me this error:
gob: encoded unsigned integer out of range
I think the issue is with the first two lines of the decoder program. So what should I put to properly decode the gob?
EDIT:
What I want is a .UnString() method for the gobString. How can i achieve that?
The encoding/gob generates binary data from Go values. The result is not for textual representation, so you should not treat it as a string, but as a series of bytes, e.g. []byte.
That said, do not use Buffer.String() but rather Buffer.Bytes() if you must obtain the encoded data.
Here's an example encoding and decoding a string value using encoding/gob:
// ENCODE
var network bytes.Buffer
encoder := gob.NewEncoder(&network)
valueToEncode := "Hello, 世界"
if err := encoder.Encode(valueToEncode); err != nil {
panic(err)
}
gobData := network.Bytes() // Save / serialize this byte slice
// DECODE
network2 := bytes.NewBuffer(gobData)
decoder := gob.NewDecoder(network2)
var decoded string
if err := decoder.Decode(&decoded); err != nil {
panic(err)
}
fmt.PrintTln(decoded)
Output (try it on the Go Playground):
Hello, 世界
Also note that if you intend to write the encoded data into a network connection or a file, you don't need bytes.Buffer, you can directly encode to those. If you must use bytes.Buffer, you may also use its Buffer.WriteTo() method to write its contents into an io.Writer (such as a file or network connection).
Related
I need to send struct data with byte slice data type during socket communication.
type A struct {
header []byte
body []byte
}
So I wrote the following source code to convert the structure to bytes.
var a A
a.header = byte slice data...
a.body = byte slice data...
buf := new(bytes.Buffer)
binary.Write(buf, binary.BigEndian, a)
However, I get an error with the binary.Write function showing the following error:
binary.Write: invalid type main.A
I have found that fixed arrays solve the problem. But since the length of the data is constantly changing, I have to use a slice rather than a fixed array.
Is there a way to solve this problem?
If you write a variable length of byte slice, the other end would not know how many bytes it needs to read. You have to communicate the length too.
So one way to send a byte slice is to first write the length (number of bytes) using a fixed-size type, e.g. int32 or int64. Then simply write the byte slice.
For example:
var w io.Writer // This represents your connection
var a A
if err := binary.Write(w, binary.LittleEndian, int32(len(a.header))); err != nil {
// Handle error
}
if _, err := w.Write(a.header); err != nil {
// Handle error
}
You may use the same logic to send a.body too.
On the other end, this is how you could read it:
var r io.Reader // This represents your connection
var a A
var size int32
if err := binary.Read(r, binary.LittleEndian, &size); err != nil {
// Handle error
}
a.header = make([]byte, size)
if _, err := io.ReadFull(r, a.header); err != nil {
// Handle error
}
Try a working example on the Go Playground.
If you have to transfer more complex structs, consider using the encoding/gob which handles sending slices with ease. For an example and some insights, see Efficient Go serialization of struct to disk.
I am new to Golang. I am developing a service which reads bytes from remote address over TCP. The problem is that I can not change encoding of bytes I read. I want to convert the bytes I read to ISO-8859-9 string. Here is part of reading code.
conn, err := net.Dial("tcp", constant.ConnectHost+":"+constant.ConnectPort)
checkError(err)
defer conn.Close()
reader := bufio.NewReader(conn)
textproc := textproto.NewReader(reader)
bytes, err := textproc.R.ReadBytes(constant.EndTextDelimiter)
checkError(err)
msg := string(bytes[:])
Code works fine. But the encoding is different than I want. It is a problem for receiving service. Any suggestion?
charmap.ISO8859_9.NewEncoder().Bytes() function wants UTF-8 format to encode. I was getting error when I try to encode my bytes. Because my incoming bytes are in 8859-9 format and I was trying to convert them directly. First I decode the bytes to UTF-8 format. I did my process, at the end I encoded this UTF-8 bytes to ISO8859-9 unicode using encoder. Here is the new code.
//main package
bytes, err := textproc.R.ReadBytes(constant.EndTextDelimiter)
checkError(err)
msg := encoder.DecodeISO8859_9ToUTF8(bytes)
//..........
// Process that string, create struct Then convert struct to json bytes
// Then encode that bytes
json := encoder.EncodeUTF8ToISO8859_9(bytes)
//encoder package
package encoder
import "golang.org/x/text/encoding/charmap"
func DecodeISO8859_9ToUTF8(bytes []byte) string {
encoded, _ := charmap.ISO8859_9.NewDecoder().Bytes(bytes)
return string(encoded[:])
}
func EncodeUTF8ToISO8859_9(bytes []byte) string {
encoded, _ := charmap.ISO8859_9.NewEncoder().Bytes(bytes)
return string(encoded[:])
}
In my Go program I am encoding []byte data with gob
buf := new(bytes.Buffer)
enc := gob.NewEncoder(buf)
//data is []byte
buf.Reset()
enc.Encode(data)
but getting 'gob decoder attempting to decode into a non-pointer' when I am trying to decoded
buf := new(bytes.Buffer)
d := gob.NewDecoder(buf)
d.Decode(data)
log.Printf("%s", d)
Gob requires you to pass a pointer to decode.
In your case, you would do:
d.Decode(&data)
reason being, it may have to modify the slice (ie: to make it bigger, to fit the decoded array)
I'm fairly new to the Go language and having a hard time achieving the following: I'm receiving a base64 string (basically, an encoded image) and need to transform it to the binary form on the server.
func addOrUpdateUserBase64(w http.ResponseWriter, r *http.Request, params martini.Params) {
c := appengine.NewContext(r)
sDec, _ := b64.StdEncoding.DecodeString(r.Body)
...
This is not working, because DecodeString expects a string... how do I transform request.Body into a string? Any tips are very much appreciated!
Do not use base64.StdEncoding.DecodeString, instead set up a decoder directly from the r.Body
dec := base64.NewDecoder(base64.StdEncoding, r.Body)` // dec is an io.Reader
now use dec, e.g. dump to a bytes.Buffer like
buf := &bytes.Buffer{}
n, err := io.copy(buf, dec)
which will decode r.Body into buf or copy directly to a http.Response or a file.
Or use Peter's method below if keeping all in memory is okay.
func (*Encoding) Decode
func (enc *Encoding) Decode(dst, src []byte) (n int, err error)
Decode decodes src using the encoding enc. It writes at most
DecodedLen(len(src)) bytes to dst and returns the number of bytes
written. If src contains invalid base64 data, it will return the
number of bytes successfully written and CorruptInputError. New line
characters (\r and \n) are ignored.
And one more option would be just casting r.Body to a string :
//Edit, fixed the code to work with an io.Reader
import "io/ioutil"
..........
if body, err := ioutil.ReadAll(r.Body); err == nil {
sDec, _ := b64.StdEncoding.DecodeString(string(body))
}
I have an io.ReadCloser object (from an http.Response object).
What's the most efficient way to convert the entire stream to a string object?
EDIT:
Since 1.10, strings.Builder exists. Example:
buf := new(strings.Builder)
n, err := io.Copy(buf, r)
// check errors
fmt.Println(buf.String())
OUTDATED INFORMATION BELOW
The short answer is that it it will not be efficient because converting to a string requires doing a complete copy of the byte array. Here is the proper (non-efficient) way to do what you want:
buf := new(bytes.Buffer)
buf.ReadFrom(yourReader)
s := buf.String() // Does a complete copy of the bytes in the buffer.
This copy is done as a protection mechanism. Strings are immutable. If you could convert a []byte to a string, you could change the contents of the string. However, go allows you to disable the type safety mechanisms using the unsafe package. Use the unsafe package at your own risk. Hopefully the name alone is a good enough warning. Here is how I would do it using unsafe:
buf := new(bytes.Buffer)
buf.ReadFrom(yourReader)
b := buf.Bytes()
s := *(*string)(unsafe.Pointer(&b))
There we go, you have now efficiently converted your byte array to a string. Really, all this does is trick the type system into calling it a string. There are a couple caveats to this method:
There are no guarantees this will work in all go compilers. While this works with the plan-9 gc compiler, it relies on "implementation details" not mentioned in the official spec. You can not even guarantee that this will work on all architectures or not be changed in gc. In other words, this is a bad idea.
That string is mutable! If you make any calls on that buffer it will change the string. Be very careful.
My advice is to stick to the official method. Doing a copy is not that expensive and it is not worth the evils of unsafe. If the string is too large to do a copy, you should not be making it into a string.
Answers so far haven't addressed the "entire stream" part of the question. I think the good way to do this is ioutil.ReadAll. With your io.ReaderCloser named rc, I would write,
Go >= v1.16
if b, err := io.ReadAll(rc); err == nil {
return string(b)
} ...
Go <= v1.15
if b, err := ioutil.ReadAll(rc); err == nil {
return string(b)
} ...
data, _ := ioutil.ReadAll(response.Body)
fmt.Println(string(data))
func copyToString(r io.Reader) (res string, err error) {
var sb strings.Builder
if _, err = io.Copy(&sb, r); err == nil {
res = sb.String()
}
return
}
The most efficient way would be to always use []byte instead of string.
In case you need to print data received from the io.ReadCloser, the fmt package can handle []byte, but it isn't efficient because the fmt implementation will internally convert []byte to string. In order to avoid this conversion, you can implement the fmt.Formatter interface for a type like type ByteSlice []byte.
var b bytes.Buffer
b.ReadFrom(r)
// b.String()
I like the bytes.Buffer struct. I see it has ReadFrom and String methods. I've used it with a []byte but not an io.Reader.