I am new to Golang. I am developing a service which reads bytes from remote address over TCP. The problem is that I can not change encoding of bytes I read. I want to convert the bytes I read to ISO-8859-9 string. Here is part of reading code.
conn, err := net.Dial("tcp", constant.ConnectHost+":"+constant.ConnectPort)
checkError(err)
defer conn.Close()
reader := bufio.NewReader(conn)
textproc := textproto.NewReader(reader)
bytes, err := textproc.R.ReadBytes(constant.EndTextDelimiter)
checkError(err)
msg := string(bytes[:])
Code works fine. But the encoding is different than I want. It is a problem for receiving service. Any suggestion?
charmap.ISO8859_9.NewEncoder().Bytes() function wants UTF-8 format to encode. I was getting error when I try to encode my bytes. Because my incoming bytes are in 8859-9 format and I was trying to convert them directly. First I decode the bytes to UTF-8 format. I did my process, at the end I encoded this UTF-8 bytes to ISO8859-9 unicode using encoder. Here is the new code.
//main package
bytes, err := textproc.R.ReadBytes(constant.EndTextDelimiter)
checkError(err)
msg := encoder.DecodeISO8859_9ToUTF8(bytes)
//..........
// Process that string, create struct Then convert struct to json bytes
// Then encode that bytes
json := encoder.EncodeUTF8ToISO8859_9(bytes)
//encoder package
package encoder
import "golang.org/x/text/encoding/charmap"
func DecodeISO8859_9ToUTF8(bytes []byte) string {
encoded, _ := charmap.ISO8859_9.NewDecoder().Bytes(bytes)
return string(encoded[:])
}
func EncodeUTF8ToISO8859_9(bytes []byte) string {
encoded, _ := charmap.ISO8859_9.NewEncoder().Bytes(bytes)
return string(encoded[:])
}
Related
I am making a go program where I need to write a gob to a file. I used the .String() method to convert the gob to a string.
var network bytes.Buffer
encoder := gob.NewEncoder(&network)
_ = encoder.Encode(valueToEncode)
gobString := network.String()
then I will write the gob to a file, and later I will retrieve it and send it to this program:
var filebytes = []byte(file) //i think that these two lines are the issue
network := bytes.NewBuffer(filebytes)
decoder := gob.NewDecoder(network)
var decoded interface{}
_ := decoder.Decode(&decoded)
but when i run this, it gives me this error:
gob: encoded unsigned integer out of range
I think the issue is with the first two lines of the decoder program. So what should I put to properly decode the gob?
EDIT:
What I want is a .UnString() method for the gobString. How can i achieve that?
The encoding/gob generates binary data from Go values. The result is not for textual representation, so you should not treat it as a string, but as a series of bytes, e.g. []byte.
That said, do not use Buffer.String() but rather Buffer.Bytes() if you must obtain the encoded data.
Here's an example encoding and decoding a string value using encoding/gob:
// ENCODE
var network bytes.Buffer
encoder := gob.NewEncoder(&network)
valueToEncode := "Hello, 世界"
if err := encoder.Encode(valueToEncode); err != nil {
panic(err)
}
gobData := network.Bytes() // Save / serialize this byte slice
// DECODE
network2 := bytes.NewBuffer(gobData)
decoder := gob.NewDecoder(network2)
var decoded string
if err := decoder.Decode(&decoded); err != nil {
panic(err)
}
fmt.PrintTln(decoded)
Output (try it on the Go Playground):
Hello, 世界
Also note that if you intend to write the encoded data into a network connection or a file, you don't need bytes.Buffer, you can directly encode to those. If you must use bytes.Buffer, you may also use its Buffer.WriteTo() method to write its contents into an io.Writer (such as a file or network connection).
I have following code which decodes base64 and then encodes it to hex.
doc_id := "Can35qPeFkm9Xgmp9+aj3g=="
base64_decode, err := base64.StdEncoding.DecodeString(doc_id)
if err != nil {
log.Fatal("error:", err)
}
base64_decoded := fmt.Sprintf("%q", base64_decode)
fmt.Printf("base_decoded %v\n", base64_decoded)
src := []byte(base64_decoded)
fmt.Println(src)
hex_encode := make([]byte, hex.EncodedLen(len(src)))
hex.Encode(hex_encode, src)
hex_encoded := fmt.Sprintf("%s", hex_encode)
fmt.Printf("hex_encoded %v", hex_encoded)
where doc_id is base64 format.
base64_decoded is its decoded value.
I have to encode it to hex, so i pass it to src.
The problem is when i pass the identifier base64_decoded to src i get wrong when i pass in the value that base64_decoded is holding i get correct answer.
for example:
if i get base64_decoded value as "\x11z\xc0[d~\xfcK\xb1\xf8\x11z\xc0[d~"
if i pass its value which is "\x11z\xc0[d~\xfcK\xb1\xf8\x11z\xc0[d~", i get correct answer 117ac05b647efc4bb1f8117ac05b647e
if i pass the variable holding "\x11z\xc0[d~\xfcK\xb1\xf8\x11z\xc0[d~" i get wrong answer 225c7831317a5c7863305b647e5c7866634b5c7862315c7866385c7831317a5c7863305b647e22dn
Has it got something with this assignment base64_decoded := fmt.Sprintf("%q", base64_decode)
what am i doing wrong
Use the following code:
doc_id := "Can35qPeFkm9Xgmp9+aj3g=="
base64_decode, err := base64.StdEncoding.DecodeString(doc_id)
if err != nil {
log.Fatal("error:", err)
}
fmt.Println(hex.EncodeToString(base64_decode))
Directly encode the bytes in base64_decode as hex.
The call to fmt.Sprintf("%q", base64_decode) returns a string with the bytes escaped per Go's string literal rules. In the general case, the bytes in the returned string are not equal to the bytes in base64_decode. For example, the input byte 0x11 is escaped to the four bytes \x11.
what happens is that the conversion that fmt.Sprintf("%q", base64_decode) makes to base64_decode, in this process to convert some bytes in the memory are lost.
here you below I leave an example: https://play.golang.org/p/pdQBp7NsvQM
doc_id := "Can35qPeFkm9Xgmp9+aj3g=="
var_encode := b64.StdEncoding.EncodeToString([]byte(doc_id))
fmt.Println(var_encode)
var_decode , _ := b64.StdEncoding.DecodeString(var_encode)
fmt.Println(string(var_decode))
In my Go program I am encoding []byte data with gob
buf := new(bytes.Buffer)
enc := gob.NewEncoder(buf)
//data is []byte
buf.Reset()
enc.Encode(data)
but getting 'gob decoder attempting to decode into a non-pointer' when I am trying to decoded
buf := new(bytes.Buffer)
d := gob.NewDecoder(buf)
d.Decode(data)
log.Printf("%s", d)
Gob requires you to pass a pointer to decode.
In your case, you would do:
d.Decode(&data)
reason being, it may have to modify the slice (ie: to make it bigger, to fit the decoded array)
I'm using gob to send messages from client to serve and this is working, however when the server response to client this don't read from connection.
func servidor(porta int){
var site string
addr := net.UDPAddr{
Port: porta,
IP: net.ParseIP("localhost"),
}
conn, erro := net.ListenUDP("udp", &addr)
verificaErro(erro)
enc := gob.NewEncoder(conn)
dec := gob.NewDecoder(conn)
dec.Decode(&site)
enc.Encode(site)
}
func cliente(site string){
porta := "1200"
conn := ConnectToSocket("localhost:"+porta)
enc := gob.NewEncoder(conn)
dec := gob.NewDecoder(conn)
enc.Encode("Test")
dec.Decode(&site)
fmt.Println(site)
}
How can I solve this problem?
There are two issues:
UDP is a packet based protocol, but gob is expecting a stream. The encoder can call the connection Write method multiple times when encoding a value. Each write sends a packet. You probably want one packet per encoded value.
Gob streams have state. The encoder sends information about any type once and expects the decoder to remember the information. Even if the gob encoder calls Write exactly once per encoded value, packets sent by subsequent writes will not include the type information.
The fix is to encode to a buffer and send that buffer:
var buf Bytes.Buffer
if err := gob.NewEncoder(&buf).Encode(value); err != nil {
// handle error
}
_, err := c.WriteTo(buf.Bytes(), addr)
Receive to a buffer and decode from that buffer:
buf := make([]byte, 1024)
n, addr, err := c.ReadFrom(buf)
if err != nil {
// handle error
}
if err := gob.NewDecoder(bytes.NewReader(buf[:n])).Decode(&v); err != nil {
// handle error
}
I'm fairly new to the Go language and having a hard time achieving the following: I'm receiving a base64 string (basically, an encoded image) and need to transform it to the binary form on the server.
func addOrUpdateUserBase64(w http.ResponseWriter, r *http.Request, params martini.Params) {
c := appengine.NewContext(r)
sDec, _ := b64.StdEncoding.DecodeString(r.Body)
...
This is not working, because DecodeString expects a string... how do I transform request.Body into a string? Any tips are very much appreciated!
Do not use base64.StdEncoding.DecodeString, instead set up a decoder directly from the r.Body
dec := base64.NewDecoder(base64.StdEncoding, r.Body)` // dec is an io.Reader
now use dec, e.g. dump to a bytes.Buffer like
buf := &bytes.Buffer{}
n, err := io.copy(buf, dec)
which will decode r.Body into buf or copy directly to a http.Response or a file.
Or use Peter's method below if keeping all in memory is okay.
func (*Encoding) Decode
func (enc *Encoding) Decode(dst, src []byte) (n int, err error)
Decode decodes src using the encoding enc. It writes at most
DecodedLen(len(src)) bytes to dst and returns the number of bytes
written. If src contains invalid base64 data, it will return the
number of bytes successfully written and CorruptInputError. New line
characters (\r and \n) are ignored.
And one more option would be just casting r.Body to a string :
//Edit, fixed the code to work with an io.Reader
import "io/ioutil"
..........
if body, err := ioutil.ReadAll(r.Body); err == nil {
sDec, _ := b64.StdEncoding.DecodeString(string(body))
}