Why does Golang base64 to Hex produce a different encoding value than the online converter?
Original String:
ARVIN
Base64 encoded:
QVJWSU4=
Golang (base64 to hex):
51564a575355343d
Online (base64 to hex):
415256494e
package main
import (
"encoding/hex"
"fmt"
)
func main() {
base64 := "QVJWSU4="
hx := hex.EncodeToString([]byte(base64))
fmt.Println("Original String: ARVIN")
fmt.Println()
fmt.Println(base64 + " ==> " + hx)
}
You can convert the un-encoded value directly to hex:
h := hex.EncodeToString([]byte("ARVIN"))
fmt.Println(h) // prints 415256494e
Given the program starts with the base64 encoding, the program must decode the base64 string to bytes and then encode the bytes to a hex string. This is what the online tool does.
The code in the question encodes the base64 string to a hex string. It's missing the decode step.
Here's how to re-encode base64 to hex in Go:
p, err := base64.StdEncoding.DecodeString("QVJWSU4=")
if err != nil {
// handle error
}
h := hex.EncodeToString(p)
fmt.Println(h) // prints 415256494e
Run it in the playground.
Related
This question already has an answer here:
How to transfer hex strings to []byte directly in Go?
(1 answer)
Closed 12 months ago.
I have the following hex data created by converting 5 values(consists of name, numeric and date field) to TLV
0115426f627320426173656d656e74205265636f726473020f3130303032353930363730303030330314323032322d30342d32355431353a33303a30305a040a323130303130302e393905093331353031352e3135
This hex data needs to be further encoded to Base64. I wrote the below code for that
func TLVsToBase64(v string) string { // v - the TLV in hex format
encodedTLV := b64.StdEncoding.EncodeToString([]byte(v))
return encodedTLV
}
The output(which is wrong) of the aforementioned hex data is below:
MDExNTQyNmY2MjczMjA0MjYxNzM2NTZkNjU2ZTc0MjA1MjY1NjM2ZjcyNjQ3MzAyMGYzMTMwMzAzMDMyMzUzOTMwMzYzNzMwMzAzMDMwMzMwMzE0MzIzMDMyMzIyZDMwMzQyZDMyMzU1NDMxMzUzYTMzMzAzYTMwMzA1YTA0MGEzMjMxMzAzMDMxMzAzMDJlMzkzOTA1MDkzMzMxMzUzMDMxMzUyZTMxMzU=
The desired output is:
ARVCb2JzIEJhc2VtZW50IFJlY29yZHMCDzEwMDAyNTkwNjcwMDAwMwMUMjAyMi0wNC0yNVQxNTozMDowMFoECjIxMDAxMDAuOTkFCTMxNTAxNS4xNQ==
I am new to Go, so please help me to troubleshoot the issue. I might missed something
Your input is the hexadecimal representation of some data. And your expected output is not the Base64 encoding of the UTF-8 data of the hex representation, but rather the data (the bytes) the hex encoding represent, so first decode the bytes e.g. using hex.DecodeString():
func TLVsToBase64(v string) (string, error) { // v - the TLV in hex format
data, err := hex.DecodeString(v)
if err != nil {
return "", err
}
encodedTLV := base64.StdEncoding.EncodeToString(data)
return encodedTLV, nil
}
Testing it:
s := "0115426f627320426173656d656e74205265636f726473020f3130303032353930363730303030330314323032322d30342d32355431353a33303a30305a040a323130303130302e393905093331353031352e3135"
fmt.Println(TLVsToBase64(s))
Output is what you expect (try it on the Go Playground):
ARVCb2JzIEJhc2VtZW50IFJlY29yZHMCDzEwMDAyNTkwNjcwMDAwMwMUMjAyMi0wNC0yNVQxNTozMDowMFoECjIxMDAxMDAuOTkFCTMxNTAxNS4xNQ== <nil>
I have a word which contains some of these characters - šđžčć. When I take the first letter out of that word, I'll have a byte, when I convert that byte into string I'll get incorrectly decoded string.
Can someone help me figure out how to decode properly the extracter letter.
This is example code:
package main
import (
"fmt"
)
func main() {
word := "ŠKOLA"
c := word[0]
fmt.Println(word, string(c)) // ŠKOLA Å
}
https://play.golang.org/p/6T2FX4vN3-U
Š is more than one byte. One method to index runes is to convert the string to []rune
c := []rune(word)[0]
https://play.golang.org/p/NBUopxe-ik1
You can also use the functions provided in the utf8 package, like utf8.DecodeRune and utf8.DecodeRuneInString to iterate over the individual codepoints in the utf8 string.
r, _ := utf8.DecodeRuneInString(word)
fmt.Println(word, string(r))
As the title say, I can find the function to give me ascii code of bytes, but not the other way around
Golang string literals are UTF-8 and since ASCII is a subset of UTF-8, and each of its characters are only 7 bits, we can easily get them as bytes by casting (e.g. bytes := []byte(str):
package main
import "fmt"
func main() {
asciiStr := "ABC"
asciiBytes := []byte(asciiStr)
fmt.Printf("OK: string=%v, bytes=%v\n", asciiStr, asciiBytes)
fmt.Printf("OK: byte(A)=%v\n", asciiBytes[0])
}
// OK: string=ABC, bytes=[65 66 67]
// OK: byte(A)=65
I tried to decode a valid (based on my understanding) base64 encoded string in Go with:
data, err := base64.StdEncoding.DecodeString(s)
if err != nil {
...
}
A full example is here. I have a string "eyJlbWFpbF9hZGRyZXNzIjoiIiwiZXhwIjoxNDQ3NzIzMzY4LCJmaXJzdG5hbWUiOiIiLCJpYXQiOjE0NDc0NjQxNjgsImlzcyI6Imh0dHA6Ly91ZGFjaXR5LmNvbSIsImtpZCI6ImE3ZTg5ZWQyMSIsImxhc3RuYW1lIjoiIiwidXNlcl9pZCI6IjEyMzQ1Njc4IiwidXNlcm5hbWUiOiJoYW5zb2xvQGhvdGguY29tIn0", which can be properly decoded for example here
or even in your browser's console with atob(that_string);, but for some reason go complains with:
illegal base64 data at input byte 236
Notice, that I can decode some other strings. So why can not I base64decode a valid encoded string in Go?
Your input does not have any padding. Therefore, you should use base64.RawStdEncoding over base64.StdEncoding:
data, err := base64.RawStdEncoding.DecodeString(s)
Example: https://play.golang.org/p/ZWfzYXQ5Ye
I'm having some trouble while reading a file which has a fixed column length format. Some columns may contain umlauts.
Umlauts seem to use 2 bytes instead of one. This is not the behaviour I was expecting. Is there any kind of function which returns a substring? Slice does not seem to work in this case.
Here's some sample code:
http://play.golang.org/p/ZJ1axy7UXe
umlautsString := "Rhön"
fmt.Println(len(umlautsString))
fmt.Println(umlautsString[0:4])
Prints:
5
Rhö
In go, a slice of a string counts bytes, not runes. This is why "Rhön"[0:3] gives you Rh and the first byte of ö.
Characters encoded in UTF-8 are represented as runes because UTF-8 encodes characters in more than one
byte (up to four bytes) to provide a bigger range of characters.
If you want to slice a string with the [] syntax, convert the string to []rune before.
Example (on play):
umlautsString := "Rhön"
runes = []rune(umlautsString)
fmt.Println(string(runes[0:3])) // Rhö
Noteworthy: This golang blog post about string representation in go.
You can convert string to []rune and work with it:
package main
import "fmt"
func main() {
umlautsString := "Rhön"
fmt.Println(len(umlautsString))
subStrRunes:= []rune(umlautsString)
fmt.Println(len(subStrRunes))
fmt.Println(string(subStrRunes[0:4]))
}
http://play.golang.org/p/__WfitzMOJ
Hope that helps!
Another option is the utf8string package:
package main
import "golang.org/x/exp/utf8string"
func main() {
s := utf8string.NewString("🧡💛💚💙💜")
// example 1
n := s.RuneCount()
println(n == 5)
// example 2
t := s.Slice(0, 2)
println(t == "🧡💛")
}
https://pkg.go.dev/golang.org/x/exp/utf8string